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This  study  focuses  on  utilizing  texture  as  a  means  of  obtaining 
description  of  surfaces  in  a  3-D  scene  from  an  image.  Surfaces  are 
assumed  to  be  planar  and  are  described  by  their  normals.  A  surface 
normal  is  parameterized  by  using  the  polar  form  of  the  gradient  space. 
I.e.  surface  tilt  and  surface  slant.  The  computational  procedure  is 
developed  assuming  statistical  homogeneity  of  frontal  views  of  textured 
surfaces.    Perspective  projection  is  chosen  as  the  imaging  model. 

Given  a  monocular  view  of  a  scene  a  two-step  procedure  for 
determining  surface  orientation  is  proposed  whereby  surface  tilt  is 
determined  and  surface  slant  estimated  from  measurable  quantities  in  an 
image.  Rotational  stereo  is  advocated  as  a  means  of  more  accurately 
determining  surface  slant.  The  proposed  methods  are  implemented  in  the 
Fourier  and  spatial  domains  and  the  obtained  results  are  very 
satisfactory. 

Boundaries  arising  from  differences  in  texture,  surface  orientation 
and  abrupt  changes  in  distance  from  the  viewer  are  identified.    A  set  of 
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constraints  arising  from  the  underlying  imaging  model  and  the  class  of 
textures  considered  is  used  to  identify  the  uniform  regions  in  an 
image.  The  boundary,  lying  in  the  intermediate  zone  between  two  uniform 
regions,  is  determined  by  using  least  squares  curve  fitting  or  linear 
recursive  filtering.  The  results  obtained  are  in  very  good  agreement 
with  human  perception. 
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CHAPTER  1 
INTRODUCTION 


Computer  vision,  a  newly  emerging  field  of  Artificial  Intelligence, 
deals  with  the  understanding  of  visual  scenes  and  the  construction  of 
effective  computer-based  visual  systems.  In  recent  years,  the  surging 
interest  in  image  understanding  for  the  tasks  of  automating  industrial 
processes,  remote  sensing  and  automatic  surveillance  has  caused  notable 
developments  in  the  extraction  o^'  3-D  information  from  images.  A  large 
number  of  approaches  for  obtaining  a  scene  description  from  an  image 
have  been  considered.  Most  frequently,  vision  research,  both  in 
computers  and  humans,  is  carried  out  in  a  rigidly  constrained 
environment  and  cues,  such  as  texture,  shading,  etc.,  are  studied  in 
isolation.  The  common  theme  of  these  pioneering  efforts  is  that  the 
stepping  stone  towards  obtaining  descriptions  convenient  for  recognition 
tasks  is  a  view-point  dependent  description  of  the  geometry  of  visible 
surfaces. 

The    study    described    in    this    dissertation    is    aimed    towards  a 

computational  theory  of  object  description  from  texture.    The  object's 

shape  is  represented  in  terms  of  planar  surfaces.     Boundaries  arising 

from   differences   in   surface   texture,    surface  orientation   or  abrupt 

changes   in  the  distance  from  the  viewer  are  identified.     This  work 

encompasses    the    computational    theories,    implementation    issues  and 

numerous   results   pertaining   to   two   types   of   problems.      The  first 

problem,    which    has    attracted    considerable    attention    in    the  past 
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[3,46,47,62,69,70,84],  deals  with  the  estimation  of  surface  orientation 
using  textural  information.  The  second  problem,  which  has  received  very 
little  attention,  is  the  determination  of  the  boundaries  between 
textured  surfaces  given  arbitrary  views  of  a  scene. 

The  main  body  of  this  dissertation  is  organized  into  four  logical 
units.  The  first  of  these.  Chapter  2,  focuses  upon  the  concept  of 
texture.  The  lack  of  a  widely  accepted  definition  of  the  term,  despite 
numerous  research  contributions  on  the  subject,  has  compelled  this 
author  to  adopt  a  fairly  personalized  style  in  treating  the  subject 
matter.  The  most  significant  contribution  in  this  part  is  the 
establishment  of  texture  taxonomy  wherein  texture  is  divided  into  five 
categories  ranging  from  ideal  to  irregular.  The  categories  are 
described  in  terms  of  tiling  theory  [16]  and  features  extracted  from 
cooccurrence  matrices  [32]. 

The  second  part.  Chapters  3  and  4,  contains  the  computational 
theory  of  3-D  scene  description  from  texture.  The  main  effort  in 
chapter  3  is  directed  toward  determining  exactly  what  information  about 
a  scene  can  be  obtained  from  an  image  using  only  the  very  basic 
assumptions  regarding  statistical  homogeneity  of  frontal  views  of 
textured  surfaces,  local  surface  planarity  and  perspective  projection. 
A  two-step  procedure  is  proposed  wherein  surface  tilt  and  surface  slant 
are  related  to  measurable  quantitites  in  an  image.  Chapter  4  contains  a 
detailed  exposition  of  the  implementation  issues.  Three  methods  are 
proposed  for  obtaining  surface  tilt.  Comparison  of  the  methods  and  the 
results  obtained  are  presented.  An  approach  to  estimating  surface  slant 
is  investigated  at  the  end  of  Chapter  4, 


The  third  unit.  Chapter  5,  concentrates  on  the  segmentation 
problem.  A  set  of  constraints,  arising  from  the  underlying  imaging 
model  and  the  class  of  textures  considered,  is  used  to  identify  the 
uniform  regions  in  an  image.  A  uniform  region  originates  from  a  planar 
surface  having  a  statistically  homogeneous  texture.  The  boundary  lying 
in  the  intermediate  zone  between  two  uniform  regions  is  approximated  by 
using  least  squares  curve  fitting  or  is  estimated  by  using  a  linear 
recursive  filter.  The  applicability  of  these  two  techniques  goes  beyond 
the  class  of  images  considered. 

Finally,  in  the  fourth  section.  Chapter  6,  a  method  for  determining 
surface  orientation  by  making  use  of  textural  information  and  rotational 
stereo  is  proposed.  Two  views  of  a  scene  constitute  a  rotational  stereo 
pair  when  the  camera  undergoes  rotation  between  two  exposures.  The 
method  relies  solely  on  geometrical  considerations  regarding  the  class 
of  textures  considered  and  the  imaging  process.  The  approach,  although 
somewhat  unorthodox,  illustrates  well  that  computer  vision  can  take 
advantage  of  situations  which  are  beyond  the  scope  of  biological  vision 
theories.  The  method  is  very  simple  and  avoids  the  inherently  difficult 
problem  of  establishing  the  correspondence  between  points  in  the  two 
stereo  images.  Most  importantly,  in  comparison  to  the  case  where  a 
monocular  view  of  a  scene  is  available,  the  method  yields  significantly 
more  accurate  results. 

In  summary,  contributions  of  this  work  are  both  of  a  theoretical 
and  an  implementational  nature.  The  computational  theories  for 
obtaining  3-D  scene  description  from  an  image  or  a  rotational  stereo 
pair  of  images  are  developed.  Moreover,  actual  implementations  are 
carried  out.  Examples  of  the  obtained  scene  descriptions  are  shown  in 
Figures  1  and  2. 
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Figure  1.    An  example  of  the  segmented  image  wherein  textured 
surfaces  are  characterized  by  different  orientation, 
intensity  and  structural  properties:  (a)  segmented 
image,  (b)  surface  orientation  relative  to  the  image. 


Tilt(rad)  Slant(rad) 

A 

Parallel  to  the  image 

B 

1.57+  Tr/2  -.785 

C 
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Does  not  exhibit  textural 

properties 

(b) 
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Figure  2.    An  example 
image,  (b) 


of  a  scene  description: (a)  segmented 
surface  orientation  relative  to  the  image. 


Tilt  (rad) 

Slant  (rad) 

A 
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.948 
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(b) 


Figure  2  —Continued. 


CHAPTER  2 
TEXTURE  MODELING  AND  TAXONOMY 

In  many  cases,  objects  in  an  Image  do  not  appear  as  clearly  defined 
regions  of  contrasting  brightness.  Instead,  they  are  perceived  as 
regions  having  distinct  texture  properties.  Texture  and  brightness  are 
the  two  most  fundamental  features.  In  monochromatic  images,  brightness 
takes  on  varying  shades  of  gray  and  a  particular  spatial  distribution  of 
gray  tones  in  termed  texture.  Therefore,  texture  and  brightness  do  not 
constitute  two  independent  concepts,  rather  they  are  inextricably 
related, 

Textural  features  are  essential  in  many  image  processing  tasks  and 
can  serve  as  a  basis  for  classifying  parts  of  an  image.  As  a 
consequence,  texture  processing  has  captured  the  attention  of  many 
researchers  ([32]  is  the  most  recent  survey).  However,  the  notion  of 
texture  is  yet  to  be  precisely  defined.  Texture  is  usually  considered 
as  a  large  number  of  elementary  patterns  densely  and  evenly  arranged 
over  a  field  of  view  [35,64].  Therefore,  texture  can  be  delineated  in 
terms  of  (1)  identifiable  elementary  patterns  called  texels  (texture 
elements)  and  (2)  placement  rules  that  govern  the  arrangement  of 
texels.  Elementary  patterns  are  usually  alike.  Criteria  for 
determining  likeness  among  elementary  patterns  may  vary.  However,  the 
following  three  are  the  most  fundamental  criteria:  (1)  intensity 
distribution,  (2)  pattern  shape  and  (3)  pattern  size. 
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An  elementary  pattern  may  be  a  single  pixel  or  an  image 
subregion.  It  should  be  emphasized  that  the  elementary  patterns  do  not 
necessarily  correspond  to  the  identifiable  perceptual  units  of 
texture.  If  overlap  is  allowed  it  is  not  an  easy  task  to  determine  what 
the  elementary  patterns  are  or  how  they  are  placed.  A  placement  rule 
involves  a  decision  making  process  as  to  where  particular  elements 
should  be  located.  The  decision  may  be  performed  deterministically  or 
probabilistically.  Different  placement  rules  do  not  necessarily 
generate  visually  distinct  textures. 

Texture  may  be  classified  as  either  ideal  or  real-world.  An  ideal 
texture  pattern  consists  of  identical  texels  repeated  by  a  perfect 
replication  process.  Usually,  texture  elements  take  the  form  of  dots, 
stars  of  line  segments  placed  against  a  contrasting  background.  Real- 
world  textures  are  frequently  associated  with  natural  scenes,  e.g. 
grass,  leaves,  rocks,  etc.  They  are  characterized  by  like  (in  terms  of 
the  three  mentioned  criteria)  elementary  patterns  experiencing  placement 
shifts  relative  to  a  perfect  arrangement.  Consequently,  real-world 
textures  can  be  treated  as  distortions  of  ideal  ones.  Alternatively, 
texture  categorization  may  be  based  on  texel  size.  Textures  containing 
small  elementary  patterns  are  known  as  microtextures  while  large  texture 
elements  constitute  macrotextures.  Another  approach  to  texture 
classification  involves  a  constituent  to  surface  relationship.  If  there 
exists  a  surface  such  that  all  texels  lie  fully  on  it,  the  texture  is 
categorized  as  painted;  otherwise,  it  is  considered  to  be  a  pointed 
texture. 

Of  primary  interest  to  this  work  are  the  digitized  monochromatic 
images  of  textured  surfaces  which  are  simply  referred  to  as  texture  in 
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the  remaining  portion  of  this  chapter.  In  general,  a  digital  image  I  of 
NxM  pixels,  in  our  notation  I(NxM),  consists  of  perceptually  identi- 
fiable regions  Ri,R2.....Rq.  i.e.  I=RiUR2U. . .URq{R^- ]Rj=(t),i?«j ).  A  region 
R^-  may  be  a  single  pixel  or  it  may  be  an  image  subregion  of  size  r  x  s, 
1  <  r  <  N,  1  <  s  <  M.  In  texture  processing,  regions  R^-,  1=1, 2, ...,q 
potentially  belong  to  p  texture  categories  0^,02,..., Cp.  Broadly 
stated,  the  goals  of  texture  processing  are  texture  segmentation  and 
categorization.  A  texture  category  is  described  by  a  feature  pattern 
vector.  Texture  categorization  Involves  a  decision  rule  which  assigns 
to  each  region  R^-  a  category  Cj  based  on  the  feature  pattern  vector. 
Depending  on  the  nature  of  texture  categories  as  well  as  their  number, 
complicated  nonlinear  discriminant  functions  may  be  necessary.  From  the 
recognition  point  of  view,  it  may  be  beneficial  to  map  the  original 
feature  space  into  a  transformed  one  and  implement  a  decision  rule  in 
the  latter. 

Various  approaches  to  texture  modeling  and  processing  are  the 
subject  to  this  chapter.  The  emphasis  is  on  the  model  arising  from 
tiling  theory  and  statistical  analysis  based  on  cooccurrence  matrices 
since  the  two  are  used  in  this  study  to  establish  the  texture  'taxonomy 
proposed  in  section  2.4. 

2.1    Texture  Modeling 

When  looking  for  formal  means  to  describe  ideal  textures  the 
classical  theory  of  regular  figures  [22]  offers  the  appropriate 
mathematical  tools.  A  regular  arrangement  of  texture  primitives 
(texels)  assumes  that  each  primitive  can  be  carried  into  any  other  one 
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by  a  congruent  transformation  or  isometry  (distance  preserving  mapping) 
leaving  the  whole  configuration  unchanged.  The  totality  of  all 
isometries  of  a  texture  pattern  constitutes  a  group.  The  notion  of  this 
group  is  a  powerful  tool  when  studying  the  class  of  ideal  textures. 
Examples  of  particular  isometries:  translations,  rotations,  and 
translations  combined  with  reflections  are  shown  in  Figures  3(a),  3(b), 
and  3(c)  respectively. 

Since  the  ideal  textures  constitute  a  small  class  of  textures  that 
are  of  interest  for  synthesis  and  analysis,  attempts  have  been  made  to 
develop  models  which  incorporate  other  texture  categories.  Zucker  [87] 
has  proposed  texture  model  which  incorporates  real  world  textures.  The 
underlying  idea  is  that  texture  can  be  described  and  analyzed  in  three 
stages.  First,  it  is  necessary  to  concentrate  on  primitives.  Then, 
ideal  textures  can  be  easily  described  by  utilizing  placement  rules. 
Finally,  a  natural  texture  can  be  described  as  deformation  of  an  ideal 
case.  Therefore,  the  following  three  concepts  need  to  be  defined:  (1) 
texture  primitives,  (2)  ideal  texture,  and  (3)  natural  texture.  A 
texture  primitive  is  defined  to  be  the  fundamental  building  block  of  an 
ideal  texture.  The  places  where  primitives  are  concatenated  are  the 
active  sites.  An  ideal  texture  pattern  is  generated  by  concatenating 
recursively  the  primitives  until  all  the  active  sites  are  filled.  This 
procedure  resembles  covering  the  plane  with  nonoverlapping  tiles  until 
no  holes  remain.  Ideal  textures  are  modeled  by  a  graph  structure.  In 
this  way,  the  notion  of  texture  primitives  is  extended  beyond  regular 
polygons.  The  vertices  of  the  graph  correspond  to  primitives  and  edges 
represent  relations  between  primitives. 
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Figure  3. 


Examples  of  symmetry  groups  involving:  (a)  translations, 
(b)  rotations,  (c)  translations  and  reflections. 
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Natural  textures  are  modeled  by  defining  the  transformations  a  from 
an  ideal  texture  graph,  IT,  to  the  graph  of  a  natural  texture,  NT.  NT 
is  a  labeled  graph  having  the  same  structure  as  IT.  Labeling  is 
necessary  since  natural  textures  are  not,  in  general,  perfectly 
regular.  An  example  of  a  transformation  a  applied  to  an  ideal  square 
grid  texture  is  shown  in  Figure  4.  The  transformation  o  relates  each 
vertex  of  IT  to  a  corresponding  vertex  in  NT,  with  preservation  of 
edges.  Square  primitives  are  mapped  into  trapezoids  by  projection  rules 
and  the  size  of  each  trapezoid  is  determined  by  its  position  in  the 
image.  The  global  transformation  a  can  be  considered  as  an  array  of 
local  transformations  (t^^p)  where  each  element  maps  a  square  into  a 
trapezoid.    The  value  of  each  t^^p  depends  on  its  position  (m,n). 

Conners  and  Harlow  [16]  have  developed  a  texture  model  which  is 
based  on  similar  mathematical  foundations  and  which  can  be  used  to 
measure  visually  perceivable  characteristics  of  texture.  Since  the 
texture  taxonomy  proposed  in  this  work  relies  heavily  on  this  model  it 
is  presented  in  a  greater  detail  in  the  next  section. 

2.2   A  Model  for  Texture  Based  on  Mathematical  Tiling 

The  texture  model  proposed  by  Conners  and  Harlow  [16]  is  founded  on 
the  mathematical  theory  of  tiling.  Similar  to  the  work  of  Zucker  [87], 
it  involves  the  following  three  components:  (1)  texture  primitives,  (2) 
ideal  texture  and  (3)  real -world  texture.  Definitions  of  these 
components  are  based  on  the  notion  of  plane  tiling.  A  plane  tiling  is  a 
family  of  closed  topological  disks  t=(T^ |i=l,2,. . . }  which  cover  the 
Eucledean  plane  without  holes  or  overlaps.    A  closed  topological  disk  T^ 
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Figure  4.    Perspective  transformation  and  labeling. 
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is  called  a  tile.  The  model  proposed  by  Conners  and  Harlow  is  of 
interest  for  our  work  because  it  formalizes  concepts  of  texture  which 
are  usually  intuitively  understood.  Moreover,  this  model  simplifies 
texture  analysis  by  introducing  the  equivalent  parallelogram  tiling. 
Only  the  very  basic  definitions  and  the  theorem  regarding  equivalent 
parallelogram  patterns  are  stated  in  this  section  while  the  details  are 
given  in  [16]. 
Primitives 

Definition  1:  A  primitive  is  a  set  (T,f)  where  T  is  a  tile  and  f 
is  a  function  which  maps  the  points  of  T  into  a  bounded  subset  of  the 
nonnegative  real  numbers  [16]. 

Definition  2:  L  is  called  a  set  of  prototiles  if  there  exists  a 
tiling  t  such  that  every  tile  in  t  is  congruent  either  directly  or 
reflectively  to  a  tile  in  z.  An  element  of  E  is  called  a  prototile 
[16]. 

Definition  3:  A  set  Sp  =  {(Ti,fi),(T2,f2),...,(Tp,fn),...}  is  an 
admissible  set  of  primitives  if  the  set  {Tj,T2,. ..!„,...}  of  prototiles 
admits  at  least  one  tiling  of  the  plane  [16], 

Definition  4:    If  Sp  is  an  admissible  set  of  primitives,  then  an 
element  {Tj,fj}  of  Sp  is  called  a  unit  pattern  [16], 
Ideal  Texture 

Definition  5:  An  ideal  texture  is  a  set  Ij  =  {Sp,Rp}  where  Sp  is 
an  admissible  set  of  primitives  and  Rp  is  an  admissible  set  of  placement 
rules  [16]. 

Definition  6:  A  placement  rule  of  a  tile  T  is  an  isometry  which 
maps  T  onto  E^,  where      denotes  the  2-0  Euclidean  space  [16]. 
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Definition  7:  A  texture  pattern  is  called  periodic  if  its  set  of 
admissible  placement  rules,  Rp,  contains  at  least  two  translation 
isometries  in  nonparallel  directions. 

Theorem  1:  Given  any  periodic  texture  g(x)  (x  =  (x,y))  there 
exists  a  parallelogram  tiling  Xp  and  a  function  of  4>  defined  on  the 
prototile  p  of  Tp  such  that  for  any  T^ctp  and  xeT^-  g(x)  =  (t)(a"^(x)), 
where  o  is  the  isometry  which  maps  p  onto  T^-  [16], 

This  theorem  allows  the  description  of  an  ideal  texture  pattern  in 
terms  of  equivalent  parallelograms  instead  of  particular  tesselations, 
e.g.  tesselations  in  Figure  5  can  be  analyzed  in  terms  of  the  shown 
equivalent  paral leograms.  It  is  worth  noticing  that  in  the  case 
involving  complex  texture  patterns  which  are  generated  by  rotations  or 
reflections  of  the  elementary  subpatterns  the  introduction  of  virtual 
texels,  i.e.  patterns  associated  with  the  equivalent  parallelograms 
simplifies  the  description. 
Real  World  Texture 

An  ideal  texture  is  transformed  into  a  real  world  texture  by  a 
particular  transformation. 

2.3   Texture  Analysis 

Texture  models  offer  a  convenient  means  for  texture  synthesis. 
However,  the  problem  of  texture  analysis  is  considerably  more 
complicated.  Two  distinct,  approaches,  statistical  and  structural  have 
been  considered.  On  the  statistical  level,  a  texture  pattern  is 
described  by  a  set  of  statistical  measures  extracted  from  a  large 
ensemble  of  local  image  properties.    On  the  structural  level,  a  texture 
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Figure  5.    Equivalent  parallelogram  tiling  of  two  periodic  tilings. 
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is  described  in  terms  of  elementary  patterns  which  occur  repeatedly 
according  to  certain  placement  rules.  The  actual  differences  between 
the  structural  and  the  statistical  approaches  are  not  clear-cut  and  many 
techniques  of  texture  analysis  (e.g.  texture  edgeness  [65,71])  cannot  be 
clearly  classified.  In  structural  analysis  primitives  are  usually 
explicitly  defined  while  on  the  statistical  level  texture  is  described 
globally. 

2.3.1    Statistical  Analysis 

Different  approaches  involving  statistical  methods  have  been 
considered  in  texture  analysis.  Guided  by  signal  processing  techniques 
a  number  of  researchers,  e.g.  [2,10,18],  have  investigated  digital 
transforms  as  means  of  describing  texture.  Most  frequently,  the 
utilization  of  a  transform  technique  is  justified  by  the  fact  that  it  is 
easier  to  determine  spatial  relations  in  the  transform  domain  than  in 
the  spatial  domain.  The  essential  distinction  between  different 
approaches  lies  in  the  choice  of  the  orthogonal  functions  used  as  the 
basis  for  the  function  space. 

Alternatively,  texture  analysis  may  be  based  on  autocorrelation. 
Autocorrelation  measures  spatial  frequencies  and  provides  information  on 
the  dependence  of  one  pixel  on  another.  A  number  of  researchers 
[55,56,76,77]  have  applied  time  series  analysis  techniques  to  estimate 
the  parameters  of  a  texture  pattern.  In  this  approach,  the  main 
emphasis  is  on  orderly  repetitiveness  of  subpatterns  while  the  minor 
variations  are  neglected  as  due  to  the  presence  of  white  noise.  Other 
essential  statistical  approaches,  e.g.  mosaic  patterns  [68],  optical 
processing  [49],  random  field  models  [31,34,66,67],  etc.,  are  limited  to 
particular  application  areas. 
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It  has  been  widely  accepted  although  not  conclusively  proven,  that 
cooccurrence  matrices  (CM)  are  the  most  powerful  statistical  tools  in 
texture  modeling  and  analysis.  Numerous  studies,  e.g.  [18,20,33],  have 
shown  that  the  second  order  statistics  derivable  from  cooccurrence 
matrices  are  important  in  texture  discrimination.  Moreover,  Weszka  et 
al.  [83]  have  concluded  that  the  CM  approach  is  more  powerful  than  other 
methods  of  texture  discrimination. 

A  cooccurrence  matrix,  G|p^9  =  f9i,j]»  otherwise  known  as  a 
spatial  gray  level  dependence  matrix  [16]  or  a  G-matrix  [75].  The 
element  g^^j  is  the  estimated  probability  of  gray  levels  i  and  j 
appearing  at  an  intersample  spacing  distance  r  in  the  angular  direction 
e  measured  counterclockwise  from  the  x-axis  as  shown  in  Figure  6.  A 
number  of  features  extracted  from  cooccurrence  matrices  that  are  useful 
have  been  considered  in  the  literature.  The  following  five  are  the  most 
common: 


Energy 


(2.1) 


Entropy 


H(G 


(2.2) 


Correlation     C(G|^^3)  =  J|J  N|l  {i-M^)(j-My)  g^^j/a^a^  ; 


(2.3) 


Local 

Homogeneity  L(G 
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Inertia 
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where  N  is  the  number  of  gray  levels 
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These  features  relate  to  texture  directionality,  coarseness,  contrast 
and  homogeneity  on  the  perceptual  level.  Julesz  [39,40]  has 
experimentally  demonstrated  that  humans  discriminate  on  the  bases  of  the 
size,  shape  and  orientation  of  contiguous  like  elements.  Changes  In 
arrangement  that  result  in  changes  in  third  or  higher  order  statistics. 
I.e.  the  changes  in  relationships  among  non-adjacent  elements,  do  not 
result  in  an  immediate  impression  upon  the  subject.  For  this  reason  the 
extraction  of  features  from  the  cooccurrence  matrices  is  usually  limited 
to  the  second-order  statistics. 

The  idea  of  cooccurrence  matrices  has  been  extended  beyond  gray 
levels.  A  generalized  form  of  cooccurrence  matrices  has  been  proposed 
[18]  whereby  the  gray  level  is  replaced  by  other  texture  properties. 
Since  a  number  of  candidate  properties  and  their  spatial  relations  exist 
in  an  image,  Zucker  and  Terzopoulos  [88]  have  developed  a  statistical 
approach  to  finding  those  spatial  relations  that  best  capture  the 
structure  of  texture  when  the  cooccurrence  matrix  representation  is 
used.  However,  most  frequently,  heuristics  have  been  employed  when 
selecting    the    appropriate    properties.       Weszka    et    al.    [83]  have 
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considered  the  use  of  matrix  wherein  the  individual  gray  levels  are 
replaced  by  the  gray  levels  taken  over  rectangular  neighborhoods  of  a 
certain  size.  Dyer  et  al.  [20]  have,  on  the  other  hand,  replaced  gray 
levels  by  texel  edges.  Edge  cooccurrence  captures  the  boundary 
properties  of  textural  elements  along  with  their  shape  and  densities. 
However,  this  scheme  ignores  the  tonal  properties  of  texture  elements. 
Tamura  et  al.  [72]  have  extracted  from  cooccurrence  matrices  a  set  of 
measurements  compatible  with  human  perception  pertaining  to  coarseness, 
directionality,  contrast,  line-likeness,  regularity  and  roughness. 
Similarly,  Tou  and  colleagues  [75,76,78]  suggested  that  angular  second 
moment,  contrast,  entropy  and  autocorrelation  are  the  most  useful 
measures  since  they  relate  to  texture  contrast  and  homogeneity  on  the 
perceptual  level. 

The  power  of  spatial  tone  cooccurrence  approach  lies  in  the  fact 
that  the  spatial  interrelationships  of  gray  tones  in  a  texture  pattern 
are  characterized  in  such  a  way  that  the  deduced  measures  are  invariant 
under  monotonic  gray  tone  transformations. 

The  main  weakness  of  the  CM  approach  is  that  the  texture  features 
obtained  are  rather  crude  and  are  unable  to  capture  the  shape  of 
distinct  texture  elements.  From  the  practical  point  of  view,  the 
cooccurrence  matrices  are  most  useful  for  the  analysis  of  textures  which 
contain  moderately  large  elements  relative  to  the  textured  surface  and 
become  less  useful  as  the  texture  element  size  increases. 
2.3.2    Structural  Analysis  of  Texture 

Structural  analysis  is  based  on  a  view  that  alike  primitives 
appearing  in  a  regular  spatial  arrangement  constitute  a  texture 
pattern.      Since   a   particular   set  of   primitives   may  yield  texture 
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patterns  which  are  visually  distinct  the  placement  rules  must  be  taken 
into  account  when  analyzing  a  texture.  Moreover,  the  results  of 
structural  analysis  may  not  be  unique  since  a  particular  texture  pattern 
may  not  have  been  generated  by  a  unique  set  of  admissible  primitives. 
This  observation  is  obvious  even  for  the  simple  texture  pattern  shown  in 
Figure  7(a);  the  possible  admissible  primitives  are  shown  in  Figure 
7(b). 

The  basic  differences  between  the  various  approaches  to  structural 
analysis  lie  in  the  way  the  task  of  primitive  and  placement  rule 
description  is  undertaken.  Carlucci  [9]  has  suggested  a  linguistic 
approach  to  texture  description.  Similarly,  Lu  and  Fu  [50]  have  used  a 
tree  grammar  approach  to  texture  analysis.  Various  forms  of  histograms 
describing  shape  and  size  of  texels  have  been  investigated  by  Tomita  and 
colleagues  [73,74],  Tsuji  and  Tomita  [79]  and  Zucker  et  al.  [89].  On 
the  other  hand,  Ehrich  and  Foith  [21]  have  investigated  the  possibility 
of  using  relation  trees  for  texture  description. 

A  somewhat  different  approach  to  texture  analysis  based  on 
statistical  measures  has  been  proposed  by  Conners  and  Harlow  [16].  In 
view  of  the  non-uniqueness  of  elementary  patterns  and  placement  rules 
their  analyzer  finds  the  simplest  structural  unit  patterns  and  the 
simplest  placement  rules.  This  task  is  accomplished  by  utilizing  the 
texure  model  founded  in  tiling  theory  (section  2.2).  Based  on  theorem 
1,  only  parallelogram-shaped  unit  patterns  are  considered  while  the 
placement  rules  are  expressed  as  two  non-parallel  vectors,  a,b, 
corresponding  to  the  sides  of  the  equivalent  parallelograms  (Figure  8). 

The  heart  of  the  procedure  for  determining  vectors  a  and  b  is  the 
determination  of  whether  for  a  given  value  p(p>0)  there  is  any  6((Xe<Tr), 
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7.    An  ideal  texture  pattern  and  the  possible  unit  patterns 
which  can  be  used  to  generate  it. 


Figure  8.    Two  vectors  f  and  b  completely  define  the  equivalent 
parallelograms  and  their  placement  rules. 
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such  that  vector  whose  components  In  the  polar  coordinate  system  are 
p  and  e,  defines  the  translation  symmetry  of  a  particular  texture.  An 
isometry  a  is  a  symmetry  of  texture  g(x)  (x=(x,y))  if  g(x)=g(a(x))  for 
all  xeE  ,  A  translation  symmetry  is  a  symmetry  in  which  every  point 
of  the  plane  moves  through  the  same  distance  in  the  same  direction.  For 

the  predetermined  sets  of  ps  and  6s  the  procedure  is  repeated  until  two 

->■ 

non-parallel  vectors  a  and  b  are  found.  Since  for  a  given  p  it  is 
possible  to  find  many  vectors  which  define  the  translation  symmetries, 
the  procedure  chooses  the  two  vectors  which  have  the  smallest  Euclidean 
magnitude. 

The  algorithm  for  performing  the  above  described  task  involves  the 
cooccurrence  matrices  (section  2.3.1).  In  general,  given  an  ideal 
texture  pattern  with  translation  symmetry  {p,9)  its  cooccurence  matrix 
has  the  following  properties: 

1.  G|pj9  is  a  diagonal  matrix, 

2.  G|p^9  r=l,2,...,p-l  have  non-zero  off-diagonal  elements, 

^'  ''|p+n,e=^|mp+n,e'  ni,n=l,2,...,  i.e.G|p^9  is  periodic  with  period 
p.  In  terms  of  the  intertia  measure  (equation  (2.5))  an  ideal  texture 
pattern  displays  periodicity  with  period  p,  i.e.  at  multiples  of  p  the 
inertia  measure  is  zero  (Figure  9(a)).  Therefore,  by  analyzing  the 
inertia  measure  it  is  possible  to  determine  the  periodicity  of  a 
particular  texture  pattern  and,  consequently,  vectors  a  and  b  are 
described  in  terms  of  6^,82  and  pi,P2  for  which  the  inertia  measure 
displays  periodicity. 

The  real  world  texture  which   is  periodic  can  be  analyzed  in  a 

similar  way.     The  basic  difference  is  that  in  this  case  the  inertia 

measure  is  not  equal  to  zero  at  the  multiples  of  the  function  period, 
however,  it  still  exhibits  periodicity  (Figure  9(b)). 
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2.4   Texture  Taxonomy 


Since  the  notion  of  texture  encompasses  a  variety  of  spatial 
distributions  of  gray  tones  it  is  necessary  to  classify  texture  in 
particular  categories  for  the  purpose  of  this  work.  Inertia  measure 
calculated  for  the  G-matrix  (section  2.3.1)  is  conveniently  adopted  for 
this  task.  The  classification  is  limited  to  the  painted  textures  and 
their  frontal  views,  i.e.  views  in  which  image  and  the  surface  plane  are 
parallel.  The'  most  fundamental  concepts  utilized  in  establishing 
texture  taxonomy  are  texture  periodicity,  almost  periodicity  [6]  and 
aperiodicity.  Periodicity  and  almost  periodicity  are  of  the  foremost 
importance  for  this  work  since  (almost)  periodic  functions  readily  lend 
themselves  to  Fourier  analysis.  The  five  texture  categories  are 
established  as  follows. 
Ideal  Texture 

A  texture  pattern  (N  x  N)  is  ideal  if  and  only  if  (1)  There  exist 
two  angles    9^  and  92(9j?i92±Tri,i=0,l,... )  such  that  the  inertia  measures 

^^'^|r,9j^        ^^^\r  9  ^    ^""^  periodic  in  r  with  periods        and  P2  , 

respectively      and      (2)        Y(G|.         )=0,  Y(G|.      ^  )=0, 

|kp2,i|>2 

k=l,2,...,  Pj,P2''0    for    ^\r,Q  9     calculated  over  any  region 

p  X  s  where  max(2pj,2P2)<  {p,s)<N  . 
Regular  Texture 

A  texture  pattern  (N  x  N)  is  regular  if  (1)  There  exist  at  least 
two  angles     9^  and  92(9j^'92±iTT,i=0,l ,. . . )     such    that     Y(G|^  g  )  and 
Y{G|^^g^)     are    (almost)    periodic    in    r   with    (almost)    periods  of 
p,  and  Pp  ,    respectively    and    (2)      Y(G|.      .  )     and     Y(G|^     ^  ) 

^  '  |KP]^»Oj  |KP2,92 

k=l,2,...,    p,  .p-^'O   are  the  local  minima  of  v^r        ^  and 

1    2  ^^^|r.9,)  Y(G|r,92) 
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respectively    when     calculated     over     any     region     p     x    s  where 
max(2Pj,2p2)<p,s<N  . 
Globally  Regular  Texture 

A  texture  pattern  (N  x  N)  is  globally  regular  if  (1)  There  exist  at 
least  two  angles  and  92(9j^t92±  iT:,i=  0,1,2...)  Pj^,p2''0  such  that 
Y{G|^  g  )  and  Y(G|^  g  )  are  (almost)  periodic  in  r  with  (almost) 
periods  of  Pj  and  Pg  ,  respectively  and  (2)  ^^^jkp  e  ^  '""^ 
k=l,2,...  are  the  local  minima  of  KGq  (r))  and  I(Gq  (r))  when 
calculated  over  region  N  x  N,  i.e.  over  the  whole  image. 
Locally  Regular  Texture 

A  texture  pattern    I  (N  x  N),I=^|jR^.,     is  locally  regular  if  (1) 
Subregion      R^(pxs)  p,s>l  i=l,2,..k,       is     globally     regular,  for 
9^(^-),  92(^')  and  p^(^'),P2(^')    and     9/^).9/j)or      B^^'Ke^^^^  or 
P^^^^^Pj^J^  or  P2^^^*P2^^^        i,j=l,2..k,i*j  and  (2)     (p^^^' ^9^^^  ^) 


^*^2^^^'^2^^'^^     are   the   vectors   of   translation   symmetry   having  the 
smallest  Euclidean  magnitude. 
Aperiodic  Texture 

A  texture  pattern  (N  x  N)  is  aperiodic  if  it  is  not  ideal,  regular, 
globally  regular  or  locally  regular. 

The  above  definitions  infer  that  all  ideal  textures  are  regular  and 
that  all  regular  textures  are  globally  regular.  Examples  pertaining  to 
the  defined  categories  and  their  corresponding  intertia  measures 
(calculated  over  the  whole  image  region)  are  shown  in  Figures  10(a)-(e). 
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Figure  10.    Texture  categories  and  their  horizontal  and  vertical 

inertia:  (a)  ideal,  (b)  regular,  v(c)  globally  regular, 
(d)  locally  regular,  (e)  aperiodic. 
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Figure  10~Continued. 
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CHAPTER  3 
SCENE  DESCRIPTION  FROM  TEXTURE 


The  central  task  in  relating  an  image  to  a  real  world  scene  is  the 
recovery  of  3-dimensional  information  from  2-dimensional  perspective 
transformation  inherently  associated  with  an  image.  In  this  work,  an 
image  is  mapped  into  a  set  of  3-D  constituents  which  mtght  have  been  its 
source,  by  making  use  of  projective  geometry,  formal  description  of 
textures,  and  assumptions  on  the  mathematical  relations  between 
observable  changes  in  a  texture  pattern  and  a  surface  layout.  Surfaces 
are  regarded  to  be  the  fundamental  elements  in  a  scene  description. 
I.e.,  the  shape  of  objects  in  a  scene  is  described  in  terms  of  the 
object's  visible  surfaces.  Textured  surfaces  belonging  to  categories  1- 
3  of  the  proposed  texture  taxonomy  are  primarily  the  subject  of  this 
research.  No  a  priori  knowledge  on  the  particular  texture  pattern  is 
assumed. 

The   objective   in   this   chapter   is    to   develop   a  computational 

procedure  for  3-D  scene  description  from  texture.    The  organization  of 

the  material  is  as  follows.    Section  3.1  contains  historical  information 

on  texture  and  other  important  cues  which  may  provide  information  on 

surface  layout.    The  set  of  assumptions  used  in  this  work  is  presented 

in  section  3.2.    Representation  of  the  surface  orientation  is  discussed 

in  section  3.3.     A  method  for  obtaining  surface  orientation  from  an 

image  via  texture  is  developed  in  section  3.4  while  the  computational 

procedure  is  outlined  in  section  3.5.  Implementation  issues  and  results 
are  presented  in  the  next  chapter. 
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3.1    Historical  Background 

Research  in  the  field  of  the  psychology  of  visual  perception  has 
provided  invaluable  insight  into  computer  vision.  In  this  respect, 
major  developments  are  ascribed  to  Gibson  [24,25]  whose  studies, 
although  erroneous  at  times,  have  stimulated  a  number  of  researchers  to 
develop  computer-based  theories  of  surface  perception.  Gibson  noted 
some  23  distinct  cues  which  play  important  roles  in  determining  surface 
layout.  While  controversy  regarding  the  source  of  information  on  an 
object's  shape  still  exists  in  psychology,  it  appears  irrefutable  that 
cues  such  as  stereopsis,  motion,  shading,  and  texture  are  important. 
Consequently,  the  research  in  computer  vision  has  focused  on  these 
topics. 

Stereopsis  and  perception  of  visual  motion  are  examples  of  multiple 
views  of  a  scene.  The  unique  solution  to  the  scene  reconstruction 
problem  can  not  be  guaranteed  even  with  multiple  views  and  the  proposed 
methods  necessarily  embody  a  set  of  assumptions.  The  essence  of 
stereopsis  is  the  computation  of  depth  information  from  disparity  in 
Images  received  by  two  eyes.  A  computational  theory  of  stereopsis  has 
been  developed  by  Marr  and  Poggio  [54]  and  its  implementation  was 
attempted  by  Grimpson  [28]  and  independently  by  Moravec  [59,60].  The 
interpretation  of  visual  motion  involves  extraction  of  information  about 
the  objects,  their  3-dimensional  shape,  and  their  motion  through  space 
on  the  basis  of  changing  images.  On  the  psychological  side,  the  problem 
was  investigated  early  by  Miles  [58]  and  Wallach  and  O'Connell  [81]. 
The  corresponding  computational  theory  emerged  from  the  pioneering  work 
of  Oilman  [80]. 
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The  surface  orientation  may  directly  be  deduced,  in  some  cases, 
from  image  intensity  values.  The  relationships  between  shape  and  the 
intensity  at  a  point  is  expressed  by  an  image  irradiance  equation. 
Mathematically,  it  is  a  non-linear  first  order  partial  differential 
equation.  The  problem  is  intrinsically  difficult  since  the  intensity 
values  do  not  depend  solely  on  surface  orientation  but  on  surface 
illumination  and  reflectance  as  well.  The  complexity  of  outdoor  scenes 
and  the  intricacy  arising  from  secondary  illumination  of  indoor  scenes 
preclude  usage  of  this  method  except  in  the  simplest  case  of  a  single 
distant  light  source  in  laboratory  conditions.  Horn  [36]  and  Horn  and 
Sjoberg  [37]  have  sucessfully  solved  this  problem  using  the  notion  of  a 
reflectance  map.  Photometric  stereo  arising  from  different  illumination 
sources  of  a  scene  offers  an  elegant  and  accurate  alternative  to 
deducing  shape  from  shading. 

Besides  texture,  which  is  discussed  in  greater  detail  in  the 
remaining  part  of  this  section,  other  more  "exotic"  alternatives  to  3-D 
scene  description  from  Images  have  been  considered.  Marr  [52,53]  has 
investigated  possibilities  of  obtaining  an  object's  shape  from  its 
contours  while  Stevens  [69,70]  concentrated  on  analyzing  surface  con- 
tours. A  qualitative  shape  recovery,  incorporating  shadows,  has  been 
proposed  by  Waltz  [82]  founded  on  the  basic  research  of  Huffman  [38]  and 
Clowes  [14],  Kanade  [43,44]  considered  the  3-D  scene  reconstruction 
problem  in  the  framework  of  the  Origami  world. 

The  role  of  texture  in  obtaining  information  about  the  shape  of 
visible  surfaces  has  attracted  considerable  attention.  Individual 
texture  elements  appear  smaller  as  the  surface  recedes  from  the  viewer 
and  are   compressed   in   the  direction   of   surface   inclination.  The 
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arrangement  of  the  elements  in  the  image  is  a  function  of  the  position 
of  the  surface  plane  relative  to  the  image  plane.  On  the  other  hand, 
compression  is  a  function  of  the  angle  between  the  two  planes.  The 
interest  in  using  these  observations  in  developing  an  appropriate 
computational  theory  is  due  primarily  to  the  fact  that  texture  is 
present  in  abundance  in  all  types  of  scenes.  In  addition,  texture 
provides  a  means  of  developing  a  relatively  "model  free"  approach  to 
scene  analysis. 

The  importance  of  texture  in  recovering  surface  orientation  was 
first  investigated  by  psychologists.  Several  researchers  have  inquired 
into  the  role  of  texture  in  determining  scene  depth  and  surface  slant 
[29].  In  particular,  they  have  concentrated  on  the  effects  of  the 
changes  in  shape,  size  and  spacing  of  texels  on  space  perception. 
Gibson  [24,25]  was  the  first  to  postulate  that  texture  carries  suf- 
ficient information  for  a  human  to  perceive  surface  orientation.  He 
hypothesized  that  the  texture  density  gradient  offers  the  primary  basis 
for  surface  perception  by  humans.  Moreover,  he  argued  that  texture 
plays  a  major  role  in  providing  a  scale  for  visual  space  perception 
[29].  Gibson's  line  of  research  has  been  pursued  by  others  {e.g. 
[3,69,70])  who  tried  to  relate  mathematically  the  notion  of  texture 
density  with  surface  orientation. 

The  first  attempt  to  develop  a  computational  theory  relating 
texture  to  surface  orientation  is  due  to  Bajcsy  and  Liberman  [3],  Their 
method  concentrates  on  measuring  texture  gradients  which  are  further 
interpreted  as  a  depth  cue.  The  method  is  only  applicable  to  longi- 
tudinal surfaces  and  suffers  from  a  number  of  shortcomings  due  to  the 
choice  of  an  incomplete  scene  model.    Nevertheless,  this  research  effort 
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was  an  important  contribution  and  impacted  further  developments  in 
obtaning  3-D  scene  interpretation  from  texture, 

Stevens  [59,70]  treated  the  problem  of  texture  gradient  perception 
in  a  more  direct  manner  by  decomposing  its  solution  into  a  2-step 
procedure.  First,  surface  tilt  is  determined  by  searching  for  texture 
"regularity."  Second,  the  functional  relation  between  the  gradient  of  a 
particular  texture  property  and  the  surface  orientation  is  used  to 
determine  surface  slant.  Unfortunately,  the  notions  of  texture 
regularity  and  texture  gradients  have  not  been  defined.  In  addition,  no 
attempt  has  been  made  to  implement  the  2-step  procedure.  Moreover,  his 
image  model  involves  orthogonal  projection  and,  therefore,  is  not 
realistic.  The  main  strength  of  Stevens'  approach  is  its  striking 
simplicity. 

Kender  [46,47]  made  an  attempt  to  determine  shape  from  texture 
within  the  framework  of  the  general  problem  solving  strategy  "shape 
from."  A  fundamental  tool  in  his  approach  is  the  normalized  textural 
property  map  (NTPM).  Intuitively,  an  NTPM  relates  a  given  2-D  image 
element  to  a  small  class  of  3-D  constituents  which  may  have  been  its 
source  in  a  scene.  It  is  actually  an  abstract  representation  of  the 
observed  effects  of  surface  orientation  on  a  particular  texture 
property.  Kender's  approach  relies  on  prestored  information  about  a 
particular  texture  pattern.  Moreover,  in  his  work  the  concept  of 
texture  element  is  very  loosely  defined  and  his  method  actually  uses 
only  an  individual  texture  element  while  it  neglects  the  repetition 
intrinsic  to  texture. 

A  more  recent  attempt  to  relate  texture  to  surface  layout  is  due  to 
Witkin  [84].    He  implemented  a  type  of  maximum  likelihood  estimator  for 
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surface  orientation  and  tested  the  algorithm  on  geographic  data  and 
natural  texture  images  showing  good  results.  The  method  employs 
orthographic  projection;  in  addition,  the  generality  of  the  model  is 
restricted  by  the  assumptions  made  on  the  image  forming  process.  More 
Importantly,  the  method  can  handle  only  one  textured  surface  in  a  scene 
since  no  consideration  is  given  to  distinguishing  between  different 
textures.  Just  recently  Davis  et  al.  [17]  have  elaborated  on 
theoretical  aspects  of  Witkin's  approach  aiming  at  a  more  powerful 
extension  of  this  work. 

A  number  of  papers  [5,11,30,57]  have  appeared  on  the  simpler 
problem  of  determining  surface  shape  from  projections  of  a  known  texture 
pattern  onto  a  3-D  surface.  This  research  is  of  limited  use  to  the  more 
difficult  problem  of  recovering  surface  shape  from  the  projection  of  an 
unknown  texture  pattern. 

In  this  study  two  novel  approaches  to  obtaining  surface  orientation 
from  texture  are  proposed.  The  first  one  utilizes  a  single  view  of  a 
scene  and  the  surface  orientation  is  estimated  only  approximately.  The 
second  approach  makes  use  of  a  rotational  stereo  pair  of  images  for 
accurate  estimation  of  the  surface  orientation.  In  addition,  the  more 
difficult  problem  of  multi-surface  scene  analysis  and  isolation  of  the 
uniform  regions  is  considered  in  both  cases.  It  is  worth  pointing  out 
that  the  actual  implementation  is  performed  in  such  a  way  that  no 
special  image  preprocessing  is  needed. 
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3.2  Assumptions 

A  successful  approach  to  describing  3-D  scenes  must  incorporate  the 
relations  between  the  observable  changes  in  a  projected  texture  pattern 
and  the  surface  layout.  In  addition,  it  is  necessary  to  incorporate  an 
appropriate  texture  model  into  a  computational  procedure.  The  heart  of 
the  problem  lies  with  the  discovery  of  the  minimal  set  of  assumptions 
which  allows  working  with  real  world  textures.  The  remaining  portion  of 
this  chapter  concentrates  on  these  assumptions  and  on  the  development  of 
a  computational  procedure  for  obtaining  surface  orientation  from 
texture. 

The  physics  of  the  imaging  transformation.  The  image  formation 
process  involves  projection  of  a  scene  through  a  pin-hole  lens  onto 
photographic  film  where  it  forms  an  inverted  image.  A  film-based 
rectangular  Cartesian  coordinate  system,  with  the  origin  located  at  the 
focal  point  and  the  x-y  plane  parallel  to  the  image  plane  is  chosen. 
The  imaging  rays  are  not  assumed  to  be  mutually  parallel;  therefore,  in 
contradistinction  to  previous  work  [69,70,84]  perspective  geometry  is 
the  appropriate  mathematical  tool  in  our  analysis.  It  should  be  noted 
that  for  convenience  we  ignore  the  fact  that  the  images  are  inverted  top 
to  bottom  and  left  to  right.  Naturally,  this  disregard  does  not  have  an 
effect  on  the  validity  of  our  geometric  model,  it  only  simplifies  the 
working  equations. 

Texture.  In  this  work  we  consider  painted  textures  whose  frontal 
view  is  statistically  homogeneous;  occasional  variations  in  the  size 
and/or  the  arrangement  of  texels  are  allowed.  From  the  computational 
point    of    view    local    variations    are    overcome,    in    some    cases,  by 
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performing  local  averaging  prior  to  or  after  particular  operations.  In 
terms  of  the  proposed  texture  taxonomy  emphasis  is  on  categores  1,  2, 
and  3.  Category  5  (aperiodic  textures)  is  given  some  consideration  in 
Chapter  4.  It  should  be  noted  that  pathological  cases  of  original  3-D 
texture  mimicking  or  canceling  the  properties  of  projections  are  totally 
out  of  the  scope  of  this  research. 

Surface  planarity.  The  fundamental  assumption  of  our  geometric 
model  is  that  objects  in  a  scene  consist  of  planar  surfaces. 

3.3    Representation  of  the  Surface  Orientation 

Given  an  image,  much  visually  derivable  information  is  available 
that  can  play  an  important  role  in  recognition  tasks  [27].  Therefore, 
it  is  necessary  to  adopt  a  formal  scheme  for  describing  shapes  of 
objects  in  a  scene.  Clearly,  object-centered  descriptions  with  minimal 
dependence  on  the  viewpoint  are  advantageous  for  the  recognition 
tasks.  However,  in  this  work  we  shall  concentrate  on  the  viewer- 
centered  description  in  terms  of  surfaces  and  their  orientations 
relative  to  the  viewer.  This  description  is  frequently  called  2  1/2-D 
scene  description. 

Before  developing  the  appropriate  geometric  model  it  is  necessary 
to  adopt  a  particular  mode  for  representing  a  surface  orientation. 
Since  we  have  assumed  surface  planarity  the  following  discussion 
concentrates  on  representing  the  planar  surface  orientation.  In 
general,  the  orientation  of  a  planar  surface  in  3-D  space  can  be 
described  in  terms  of  the  surface  normal  which  has  two  degrees  of 
freedom,    A  variety  of  choices  is  available  for  representing  these  two 
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variables;  in  our  case  the  basic  requirement  is  that  the  chosen  form  of 
surface  representation  is  compatible  with  the  information  that  can  be 
obtained  from  an  image. 

Given  a  surface  S  analytically  described  by 

z=ax+by+c  (3.1) 

its  orientation  relative  to  image  I  may  be  conveniently  represented  in 
the  coordinate  system  shown  in  Figure  11.    The  surface  normal 

(J=af+bj-ic  (3.2) 

projects  onto  the  x-y  plane  as 

n=af+bj  .  (3.3) 

Adoption  of  the  standard  gradient  space  notation,  i.e.  setting  a=p  and 
b=q,  allows  the  use  of  ordered  pairs    (p,q)  ,  where 


as  the  indicators  for  how  the  planes  are  slanted  relative  to  the  view 
line  (z-axis).  In  polar  coordinates,  the  pair  (a,b)  is  converted  into 
(t,s)  where 


T=tan"-^(b/a)  (3.5) 

and 

s=(a2+b2l/2.  (3.6) 
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Figure  11. 


Viewing  geometry,  surface  tilt  and  surface  slant. 
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The  physical  meaning  of  variable  s  is  easily  grasped  when  one  considers 
the  angle  between  the  image  and  the  surface  plane  given  by 


Variable  t  (equation  3,5)  is  known  as  surface  tilt  while  a  is  the 
surface  slant. 

The  two  degrees  of  freedom  of  the  surface  orientation  can  be 
expressed  as  either  (p,q)  or  (t.tana)  ,  the  relation  between  the  two 
being  given  by  equations  (3.4,3.5,3.7).  However,  the  representation  of 
the  surface  orientation  whose  slant  approaches  u/2  can  not  easily  be 
handled  with  these  variables.  Instead,  one  may  use  (T,a)  with  a  being 
the  angle  between  image  I  and  surface  S  and  t  the  angle  between  the 
projection  of  the  surface  normal  onto  I  and  the  x-axis  in  I  (Figure  11. 

Alternatively,  one  may  consider  using  the  companion  Gaussian  sphere 
[47].  The  main  advantage  of  using  Gaussian  sphere  is  that  in  the 
corresponding  notation  one  may  distinguish  the  side  of  the  surface  being 
visible.  In  addition,  surfaces  parallel  to  the  z-axis  are  handled 
without  difficulty.  However,  this  latter  advantage  does  not  have  any 
practical  interest  for  this  work.  In  general,  the  gradient  space  is 
considered  to  be  more  "natural"  [47]  and  the  associated  computations  are 
much  simpler.  Therefore,  we  choose  the  (t.a)  representation  for  our 
computational  procedure.  Equivalently,  it  is  possible  to  describe  a 
scene  in  terms  of  relative  distances  from  the  viewer  since  they  are 
directly  derivable  from  surface  orientation. 


(3.7) 


hence. 


s=tana  . 


(3.8) 
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3.4    Surface  Orientation  From  a  Monocular  View  of  a  Single  Textured 


Under  the  assumptions  outlined  In  section  3.2  a  physical  surface  S 
being  a  plane  is  described  by  a'x  +  b'y  +  c'z  +  d'=0.  Then,  the  texture 
in  an  image  corresponds  to  the  pespective  projection  of  the  texture 
lying  in  S  onto  the  image  plane  A'x  +  B'y  +  C'z  +  D'=0.  From  elementary 
Euclidean  geometry  we  know  that  there  always  exists  a  family  of  parallel 
lines  in  S  which  are  parallel  to  the  image  plane.  These  lines  have  the 
parametric  description 


When  projected  onto  the  image  plane  lines  (3.9)  remain  parallel  and  are 
called  the  characteristic  lines.  Moreover,  in  the  case  of  an  ideal 
texture  the  equidistant  points  A,B,C,  lying  along  a  line  belonging  to 
(3.9),  remain  equidistant  when  projected  onto  the  Image  plane  (Figure 
12).  Hence,  by  virtue  of  texture  periodicity  texture  in  an  image  is 
periodic  along  the  characteristic  lines.  In  the  case  of  an  ideal 
texture  pattern  texture  in  the  surface  plane  is  periodic  or  almost 
periodic  along  the  characteristic  lines.  As  a  consequence,  texture  in 
the  image  is  (almost)  periodic  along  the  characteristic  lines.  The 
angle  that  a  characteristic  line  forms  with  the  x-axis,  measured 
counterclockwise.  Is  called  the  characteristic  angle.  The  period  of  a 
texture  pattern  along  any  characteristic  line  is  the  characteristic 
periodicity.  Since  the  lines  given  by  equation  (3.9)  preserve  the 
directional  coefficients,  the  characteristic  lines  remain  Invariant 
under  the  perspective  projection.  Some  other  invariant  properties  over 


Surface 


x-y^  y-y^ 
Fc^  =  ^ 


(3.9) 
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Figure  12.    Perspective  projection. 
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perspective  projection  are  col linearity,  cross  ratio,  proximity, 
smoothness,  spatial  order  and  straightness. 

The  notion  of  the  characteristic  lines  can  be  readily  extended  to 
regular  and  globally  regular  textures.  In  these  cases  an  image  texture 
pattern  is  as  well  perodic  or  almost  periodic  along  the  characteristic 
lines.  In  addition,  the  direction  of  texture  gradient,  along  which  a 
texture  pattern  changes  the  most  where  projected  onto  the  image  plane, 
is  orthogonal  to  the  characteristic  lines. 

Having  established  the  basic  relationships  arising  from  the 
perspective  projection  we  can  proceed  with  the  development  of  the 
geometric  model.  The  purpose  of  a  geometric  model  is  to  express  the 
relationships  between  the  quantities  to  be  estimated  and  the  quantitites 
to  be  measured  in  an  image.  Given  surface  S:  z=ax+by+c  and  the  image 
plane  I:    Z=const.,  the  characteristic  line  in  I  given  by  equations 


z  =  ax+by+c  ,3 
z  =  const 


is  orthoganal  to  the  surface  normal  and  its  projection.  Therefore, 
taking  into  account  the  two  angles  that  can  be  measured  between  the 
characteristic  line  and  the  x-axis,  the  following  two  relations  exist 
between  the  surface  tilt  t  and  the  characteristic  angle  n 


Equation  (3.11)  can  be  concisely  written  as 


T=n±Tr/2  , 


(3.12) 
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Therefore,  once  the  characteristic  angle  has  been  determined  the  surface 
tilt  is  known.  The  remaining  task  of  the  geometric  model  is  to  find  the 
relationships  between  measurable  quantities  in  an  image  and  the  surface 
slant.  From  Figure  12  we  can  establish  that  distance  D  between  points 
A(Xj,yj,z^)  and  B  (x2,y2.Zi)    projects  as 

d.  =!!!pi(i4)'«.i£  (3.13)' 

^         ^1  b^  h 

where  <!)  is  the  focal  distance;  distance  D  between  points  E  and  G 
projects  as 


i.-T-  (3.14) 
^  ^2 


d^  and  d2  are  the  characteristic  periodicities  for  characteristic  lines 
Lj  and  L2  respectively.    On  the  other  hand,  distance  a  projects  as 


where 


V  cos  0-6  sin  a]  (3.15) 

^r2  ^ 

y  i^2~^i^  2 

B  =  .  (3.16) 

^2  ^1 


Equations  (3.13-3.16)  constitute  the  part  of  the  geometric  model  used 
for  determining  surface  slant.  Clearly,  in  the  case  of  a  totally 
unknown  texture  the  number  of  unknowns  (only  <j)  is  assumed  known  and 
di,d2,v  are  measured  in  the  image)  exceeds,  by  far,  the  number  of 
equations.  However,  for  a  relatively  large  textured  surface  we  can 
approximate  Zi=^2  since  Zj^=ZQ+k(A  sin  a)  and  =  ZQ+(k+l)A  sin  a  and 
equation  (3.15)  becomes 
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COS  cr  - 


6  sin  a  . 


(3.17) 


Furthermore,  since  B«z|  equation  (3.17)  becomes 


V 


cos  0 


(3.18) 


where 


(3.19) 


Finally,  we  can  estimate  surface  slant  by  rewriting  equation  (3.18)  as 


This  relationship  allows  rough  estimation  of  surface  slant  since 
quantities  ?  and  v  can  be  measured  in  an  image.  More  precise  estimation 
requires  more  than  one  view  of  a  scene.  A  rotational  stereo  using  two 
views  of  a  scene  is  discussed  in  Chapter  6. 


Equations  (3.12)  and  (3.20)  are  the  heart  of  our  approach  to 
estimating  surface  orientation  via  texture.  The  proposed  procedure 
consists  essentially  of  two  parts.  First,  surface  tilt  is  determined  by 
finding  the  characteristic  angle.  Then,  surface  slant  is  estimated  by 
measuring  the  characteristic  periodicity. 


a  ~  tan-l(ii)  . 


(3.20) 


3.5   Computational  Procedure 
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The  actual  implementation  is  divided  into  the  following  five  stages 

(1)  determination  of  the  characteristic  angle  n, 

(2)  determination  of  the  surface  tilt  (equation  3.12), 

(3)  image  (window)  rotation  by  ^, 

(4)  determination  of  the  characteristic  periodicities  d^  and  d2  and 
distance  v, 

(5)  estimation  of  the  surface  slant  (equation  3.20). 

The  methods  for  performing  these  five  tasks  are  presented  in  the  next 
chapter. 


CHAPTER  4 
COMPUTATIONAL  METHODOLOGY 


The  primary  emphasis  of  this  chapter  Is  the  Implementation  of  the 
proposed  computational  procedure  for  estimating  surface  orientation. 
The  first  part  of  the  chapter  concentrates  on  the  more  difficult  problem 
of  determining  surface  tilt;  three  methods  are  considered  and  their 
applicability  is  discussed  in  detail.  The  second  part  of  the  chapter 
deals  with  surface  slant  estimation.  Finally,  the  Issue  of  image 
rotation  is  addressed, 

4.1.    Determination  of  Surface  Tilt 

Characteristic   angle   is    the   means   of   obtaining   surface  tilt. 
Determination   of   the   characteristic  angle  may   be   performed  in  the 
spatial  or  In  a  transform  domain.     In  this  work  we  consider  both  and 
discuss  their  advantages  and  disadvantages. 
4,1,1    Analysis  in  the  Fourier  Domain 

It  is  well  known  that  the  absolute  intensity  information  may  be 
unreliable  when  working  with  textured  surfaces.  For  this  reason  we 
consider  working  in  the  Fourier  domain  as  a  more  reliable  approach  to 
determining  the  characteristic  angle  and  the  characteristic  period- 
icity. The  problem  of  texture  analysis  in  the  Fourier  domain  reduces  to 
finding  the  correspondence  between  the  energy  spectrum  of  a  texture 
pattern  and  its  observable  statistical  properties.    Fourier  domain  based 
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techniques  are  superior  to  spatial  domain  techniques  when  working  with 
real  world  textures  In  many  aspects.  Most  Importantly,  they  are  Immune 
to  the  presence  of  moderate  amounts  of  noise. 

A  two-dimensional  Fourier  transform  of  a  N  x  N  Image  region  Is 

F(u.v)  =^  JjlJ  yljl(x,y)k(x,y,u,v) 

where  k{x,y,u,v)  Is  the  transform  kernel.  The  Inverse  transform  Is 
defined  as 

Kx.y)  =  III  JJ|jF(u,v)k(x.y.u.v) 

Where  k(x,y,u,v)  Is  the  Inverse  transform  kernel.  The  transform 
kernel  takes  the  form 

k{x,y,u,v)  =  exp  {-  lji(ux  +  vy)) 
and  the  Inverse  transform  kernel  Is 

k{x,y,u,v)  =  exp  (-^(ux  +  vy)). 
In  general,  F(u,v)  Is  a  complex  function.  I.e. 

F(u.v)  =  F^(u,v)  -  1  Fj(u,v) 

where 

Fr{u,v)  =  2  ^IJ  hi  I{x,y)  cos  (^1  (ux  +  vy)) 
Fj(u.v)  =i2  III  ^1  Kx.y)  sin  (|l(ux  +  vy))  . 
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The  power  spectrum  of  I(x,y)  is  the  magnitude  of  the  complex  function 
F(u,v)  and  is  given  by 

P(u.v)  =  {F[^^(u.v)  +  Fj^(u.v)?^2. 

The  power  spectrum  is  invariant  to  translation  but  not  invariant  to 
rotation  in  the  spatial  domain  [7,45,86],  This  implies  that  the  power 
spectrum  is  useful  for  distinguishing  directional  properties  of 
texture.  For  this  purpose  it  is  convenient  to  analyze  spectrum  in  polar 
coordinates  {r,<t))  and  calculate 

P(*^)  =  %l  P(r,*^)  (4.1) 


where  N  is  the  size  of  the  square  region  under  consideration. 
Identification  of  the  significant  peaks  in  the  distribution  of  P((j)) 
provides  the  means  of  determining  directionality  of  front  views  of 
textured  surfaces,  A  peak  P((j)^)  is  considered  significant  if  P{<j)^)> 
tg  where  =  M  +1.5  s;  M  is  the  mean  and  s  is  the  standard  devia- 
tion of  P(<|))  [2,3],  The  analysis  of  images  containing  arbitrary  views 
of  textured  surfaces  is  more  complex.  Differentiation  of  the  charac- 
teristic angle  from  the  angles  corresponding  to  peaks  arising  from 
directionality  inherent  to  a  texture  pattern  is  not  always  straight- 
forward. We  found  that  the  window  size  plays  a  major  role  in 
accomplishing  this  task.  Within  a  small  window  a  periodic  texture 
pattern  exhibits  a  slow  change  along  the  direction  of  texture  gradient 
while  it  is  periodic  or  almost  periodic  along  the  characteristic 
lines.    Therefore,  due  to  the  fact  that  negligible  changes  of  a  texture 
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pattern  are  observable  along  the  direction  of  texture  gradient  small 
windows  reveal  little  about  the  characteristic  angle.  Increasing  the 
window  size  results  in  the  Introduction  of  a  larger  texture  region  and 
the  changes  along  the  direction  of  texture  gradient  are  easier 
observable.  As  a  consequence,  texture  changes  significantly  along  the 
direction  of  texture  gradient  while  it  remains  periodic  along  the 
characteristic  lines.  In  terms  of  the  power  spectrum  these  changes 
cause  changes  in  the  ratios  of  the  magnitudes  of  the  peaks  associated 
with  certain  frequencies.  In  general,  peaks  associated  with  charac- 
teristic periodicity  increase  relative  to  the  magnitude  of  the  other 
frequency  components.  Therefore,  by  increasing  the  window  size  and  by 
recording  the  changes  in  the  ratios  of  the  magnitudes  of  the  peaks  in 
P(<|»)  the  characteristic  angle  can  be  determined. 

An  example  illustrating  this  is  shown  in  Figure  13.  The  character- 
istic angle  for  the  left  side  of  the  cube  shown  in  Figure  13(a)  is 
f2=2.35rad  .  Windows  16  x  16  and  32  x  32  show  very  little  energy  for 
this  angle.  On  the  other  hand  window  64  x  64,  and  for  that  matter  any 
larger  window,  reveal  a  significant  peak  for  (t)^=2.35rad  .  Spectra 
shown  in  Figures  13(b)-13(d)  are  displayed  as  image  intensity  functions 
where  the  darkness  of  a  pixel  is  directly  proportional  to  the  magnitude 
of  the  corresponding  frequency  component.  In  order  to  display  the 
higher  frequency  components  the  function  D(u,v)=log(l  +  P(u,v))  is  used 
instead  of  P{u,v).  For  easier  interpretation,  the  origin  of  the 
spectrum  is  shifted  to  the  frequency  point  (N/2, N/2). 

P(*)  is  calculated  for  a  predetermined  set  of  angles  {0,. 124, .245, 
.322, .464,. 588, .674,. 785,. 896,. 983, 1.10, 1.24, 1.32, 1.45, 1.57. 1.69, 1.82.1. 
90, 2. 04, 2. 15, 2. 24,2. 35, 2. 46, 2. 55, 2. 67, 2. 89, 2. 96, 3.017rad}.      For  angles 
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(a) 


Figure  13. 


Effects  of  the  window  size  on  the  power  spectrum: 
(a)  image  sample,  (b)-(d)  power  spectra  of  cocentric 
windows  taken  in  the  region  of  the  left  side  of  the  cube. 
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(d) 


Figure  13 


— Continued. 
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0,TT/4,ir/2,3iT/4,  P(<}>)  is  directly  calculated  from  equation  (4.1).  For 
the  other  angles  P(<t))  is  calculated  by  considering  N/2  equidistant 
points  in  the  interval  from  (0,0)  to  the  window  edge  utilizing  bilinear 
interpolation  based  on  the  existing  discrete  values  of  the  power 
spectrum.  Finally,  the  local  maxima  of  {Po,^.124**  •'^3.017^ 
determined  in  order  to  find  the  angles  associated  with  significant  peaks 
in  the  angular  distribution  of  P.  The  spectrum  is  calculated  by  using 
the  Cooley-Tukey  FFT. 

Alternatively,  the  characteristic  angle  may  be  determined  by  using 
1-0  spectral  analysis.  In  this  case  spectra  are  calculated  in  different 
directions  and  the  analysis  aims  at  differentiating  between  the  random 
distribution  of  texels  in  the  direction  of  texture  gradient  versus  the 
regular  organization  of  texels  along  the  characteristic  lines.  In 
general,  the  spectrum  of  a  periodic  function  shows  distinct  behavior 
compared  to  the  spectrum  of  a  random  function.  In  the  latter  case, 
energy  is  divided  among  different  frequencies  and  overall  the  spectrum 
shows  a  lot  of  ripples.  On  the  other  hand,  a  periodic  function  has 
energy  accumulated  at  particular  frequencies.  Examples  of  the  1-D 
Fourier  power  spectra  calculated  along  the  characteristic  line  and  along 
the  direction  of  texture  gradient  for  the  image  shown  in  Figure  14(a) 
are  given  in  Figures  14(b)  and  14(c).  It  is  worth  noticing  that 
additional  confirmation  of  the  characteristic  angle  may  be  obtained  by 
calculating  1-D  spectra  along  consecutive  lines.  The  spectra  corres- 
ponding to  the  square  wave  functions  taken  along  the  characteristic 
lines  show  consistent  position  or  gradual  change  of  the  frequency  of  the 
first  peak. 


(a) 


Figure  14.    1-0  power  spectrum  analysis:  (a)  image  sample, 

(b)  spectrum  calculated  along  the  characteristic  line, 

(c)  spectrum  calculated  along  the  texture  gradient. 
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It  IS  Important  to  note  that  the  interpretation  of  the  Fourier 
power  spectrum  is  based  on  the  continuous  transform  while  the  actual 
implementation  is  carried  out  as  a  discrete  transform  (Fast  Fourier 
Transform).  Difficulties  appearing  in  many  of  the  applications  are  due 
to  the  differences  in  the  continuous  and  discrete  transforms.  Here,  we 
shall  briefly  mention  the  most  prominent  of  all  of  the  difficulties  met 
in  the  spectral  analysis. 

It  is  well  known  that  if  a  continuous  function  is  band  limited, 
i.e.  if  its  Fourier  transform  is  zero  for  all  frequencies  greater  than 
some  frequency  f^,  then  the  choice  of  the  Nyquist  sampling  rate  (2f^.) 
ensures  that  the  continuous  function  can  be  uniquely  determined  from  its 
sample  values  and  that  no  aliasing  is  present  in  the  spectrum  of  the 
sampled  function.  Clearly,  if  the  function  is  not  band  limited  there  is 
significant  aliasing  introduced  by  the  sampling  process.  The  only  known 
remedy  is  to  increase  the  sampling  rate.  Naturally,  available  hardware 
introduces  practical  limitations. 

The  second  point  one  needs  to  bear  in  mind  when  interpreting  the 
digital  Fourier  transform  are  the  truncation  effects.  The  most  serious 
difficulties  arise  when  the  function  is  truncated  using  a  window  width 
different  from  the  multiples  of  the  function  period.  These  difficulties 
are  known  as  leakage  and  have  impact  on  the  numerical  procedure  for 
determining  local  maxima  in  the  spectrum. 

In  summary,  we  have  considered  two  methods  for  determining  the 
characteristic  angle  in  the  transform  domain.  It  is  possible  to  use  2-D 
spectral  analysis  and  associate  peaks  in  the  angular  distribution  of 
power  spectrum  with  the  characteristic  angle.  The  corresponding 
analysis  may  not  always  be  straightforward.     In  addition,  the  actual 
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calculations  involve  a  very  large  number  of  operations;  for  the  window  N 
X  N  (N=2")  the  required  number  of  additions  a(n)  is  given  recursively  by 
a(n)=2a{n-l)+2",  a(0)=0  while  m(n)=2m(n-l)+2""l,m(0)=0  multiplications 
are  needed.  The  bilinear  interpolation  and  calculations  of  the  angular 
distribution  of  the  power  spectrum  introduce  additional  computational 
burden.  Alternatively,  one  may  option  for  1-D  spectral  analysis  in 
order  to  determine  the  characteristic  angle.  This  analysis  is  more 
convenient  from  the  computational  point  of  view  since  it  Involves  a 
smaller  number  of  operations.  The  analysis  amounts  to  finding  the  local 
maxima  in  the  1-D  spectrum  calculated  in  different  directions.  The  1-D 
spectral  analysis  is  very  convenient  for  simple  texture  patterns  con- 
taining lines,  blobs,  etc.  In  the  case  of  complex  patterns  the  analysis 
may  not  be  straightforward.  In  addition,  this  approach  requires  either 
specially  designed  hardware  or  software  to  handle  image  rotation.  The 
algorithm  for  performing  image  rotation  Is  presented  in  section  4.3. 
4.1.2    Analysis  in  the  Spatial  Domain 

Spatial  domain  analysis  is  most  conveniently  done  on  the  binary 
image  containing  extracted  texel  edges.  The  imaginary  lines  forming  an 
angle  9^  with  the  x-axis  intersect  the  extracted  edges  at  ei,e2,.,.e^+i 
(Figure  15)  and  the  distance  between  e|^  and  ei^.^  is    6^_j  .    Then,  for 

6  0  9 

any  Lg  distances    5^  ,5^  ,...,6^     can  be  measured.    The  criterion  for 

determining  the  characteristic  angle   a  can   be  established  based  on 

9      6  6 

"^k+l~"^k    •    ^"        ^^^^  °^       i(ie^i^  texture  pattern  texture  in  an 
image  is  periodic  or  almost  periodic  along  the  characteristic  lines. 
Texture  shown  in  Figure  15  is  periodic  along  the  characteristic  lines 
and  for    e=fi  ,  where    Q   is  the  characteristic  angle,  6^^=&^^=.,=s^^=v 
where  v  is  the  characteristic  periodicity.  Moreover,  di^=d2®=. .=d,-!i=0. 
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Figure  15.    An  ideal  texture  pattern. 
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For  more  complex  patterns  and  regular  or  globally  regular  textures  a=0 
for  which  s{9)=pipj.  j|J(M^-cl^^)2  ^"^TT=TT  k^l  ^k^^  has  global 
minima  since  along  characteristic  lines  texture  is  almost  periodic. 
Most  importantly,  spatial  domain  analysis  is  useful  for  some  types  of 
irregular  textures.  In  that  case  it  is  convenient  to  utilize  |d|^|  and 
combine  the  notion  of  the  characteristic  lines  with  the  notion  of  the 
direction  of  texture  gradient.  As  we  have  already  pointed  out  the  two 
are  orthogonal  to  each  other.  Therefore,  if  s{e)  has  minimum  for 
and  maximum  for  and  -^■n/2-\Q^-Q^\<e  (where  e  is  a  small  number) 
then  8j  is  the  characteristic  angle.  The  applicability  of  the  spatial 
domain  technique  to  some  Irregular  textures  is  its  major  advantage  over 
the  Fourier  domain  based  methods.  An  example  of  a  binary  image  showing 
a  pavement  which  is  very  well  suited  for  this  analysis  is  shown  in 
Figure  16. 

The  spatial  domain  analysis  is  computationally  convenient  for  the 
textures  which  are  not  characterized  by  complicated  painting  functions. 
We  refrain  from  utilizing  this  technique  more  frequently  primarily  due 
to  the  fact  that  the  extraction  of  edges  in  textures  is  not  always  a 
straightforward  nor  an  easy  task. 

4.2    Estimation  of  the  Surface  Slant 

Surface  slant  is  estimated  from  the  measurements  of  the  character- 
istic periodicities  by  using  equation  (3.20).  Prior  to  performing  the 
measurements  image  needs  to  be  rotated  in  such  a  position  that  the 
characteristic  lines  are  parallel  to  one  of  the  axes,  e.g.  the  x-axis. 
The  method  employed  for  this  purpose  is  discussed  in  detail  in  section 
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Figure  16.    An  example  of  a  binary  image  characterized  by  aperiodic 
texture. 
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The  Initial  guess  on  the  characteristic  periodicity.  In  this  work. 
Is  made  based  on  the  analysis  of  1-D  Fourier  spectrum  taken  along  the 
characteristic  lines.  The  frequency  of  the  first  peak  In  the 
spectrum  Is  known  to  correspond  to  the  period  of  the  function.  Hence, 
by  numerically  identifying  frequency  fg  one  may  roughly  estimate  the 
corresponding  characteristic  periodicity.  Due  to  the  inability  to 
choose  window  sizes  exactly  equal  to  a  multiple  of  the  function  period  T 
frequency  fg  does  not  exactly  correspond  to  T.  Therefore,  we  have 
employed  additional  processing  to  determine  accurately  the  character- 
istic periodicity.  For  this  purpose  the  cooccurrence  matrices  Gj^j  ^  and 
the  corresponding  intertia  measure  are  used.  The  Inertia  measure, 
Y(G|jj|^jj)  ,    is   determined   for    .^L^d^^     (where   N    is    the  window 

0  0 

size).  Value  d  for  which  inertia  Y(Gj^  ^)  is  minimum  is  the  multiple 
of  the  characteristic  periodicity.  Since  equation  (3.20)  requires 
measurements  of  two  characteristic  periodicities  and  the  distance 
between  the  corresponding  characteristic  lines,  it  is  convenient  to 
calculate  the  1-D  spectra  for  consecutive  characteristic  lines  and 
compare  them.  In  general,  some  of  the  spectra  are  going  to  be  different 
because  they  correspond  to  the  image  regions  which  are  gaps  or 
pecularities  of  the  texture  pattern.  Examples  of  spectra  for 
characteristic  lines  separated  by  distance  D=5  pixels  for  the  image 
shown  in  Figure  14(a)  are  shown  in  Figure  17.  Clearly,  spectra  #9  and 
#16  are  different  (they  correspond  to  the  horizontal  lines  in  the 
texture  pattern).  Since  these  spectra  correspond  to  the  same  elements  of 
a  texture  pattern  they  may  be  conveniently  chosen  for  determining  the 
characteristic  periodicities  dj  and  d2. 
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Figure  17.    1-D  spectra  taken  along  characteristic  lines  in 

intervals  D=5  for  the  image  shown  in  Figure  14(a). 
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Figure  17— Continued. 
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4.3    Image  Rotation 

The  proposed  computational  procedure  for  determining  surface  orien- 
tation is  applied  to  digitized  versions  of  the  original  image.  Due  to 
the  digitization,  complete  information  is  not  available  in  directions 
other  than  0,Tr/4,Tr/2,3Tr/4  .  Therefore,  when  the  characteristic  angle 
has  an  arbitrary  value  all  subsequent  processing  has  to  be  performed  on 
incomplete  data  sets.  For  this  reason  we  have  incorporated  step  3  into 
the  proposed  computational  procedure  (section  3.5),  By  window  rotation 
in  this  case  it  is  meant  inscription  of  the  largest  possible  square 
window  into  the  digitized  image  such  that  the  angle  between  the  basis  of 
the  two  (Figure  18)  is  equal  to  the  rotation  angle  a.  Since  the  origi- 
nal window  in  our  case  is  square  and  the  rotation  angle  0  <  a  <  tt  , 
the  inscribed  window  has  dimensions  N^xNj^  where 

N  N 
1      cos  a  +  sin  a 

and  N  is  the  size  of  the  image.    Pixel  density  is  preserved  by  placing 
NjxNi    points    into    the   window.       Gray    level    of    any    point  (u,v) 
i-1  <  u  <  i,j-l  <  V  <  j  is 

I(n,m)  if  D[(u,v),(n,m)]  =  min  D[(u,v),(T,n)]  '^'\'}'\ 

?  (u,v) 

^nl:]^'^  W^i^^  ^'7^  otherwise 

where 

D[(u,v),(n,m)]  =  ((u-n)2+(v-m)2)^2 


Figure  18.    Window  rotation. 


72 


and  n=i-l  or  n=i;  m=j-l  or  m=j.  The  square  grid  points  are  the  Image 
pixels  I(x,y).  An  example  of  a  rotated  Image  obtained  by  using  this 
algorithm  Is  shown  In  Figure  19(b);  Its  original  (digital  version)  is  in 
Figure  19(a). 
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(a) 

gure  19.    Image  rotation,  (a)  Original  image,  (b)  Rotated  image. 
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CHAPTER  5 
SEGMENTATION 


Texture  classification  and  segmentation  have  been  the  subject  of 
extensive  research  during  the  last  decade.  The  most  significant 
accomplishments  have  been  surveyed  by  Haralick  [32]  and  Rosenfeld  and 
Davis  [63].  The  early  research  efforts  concentrated  on  the  problem  of 
texture  classification,  e.g.  [26,48].  These  approaches  Involve  the 
extraction  of  an  appropriate  set  of  texture  features  and  the  training  of 
the  classifier  to  recognize  particular  textures.  Various  types  of 
classification  algorithms  have  been  evaluated  by  Weszka  et  al.  [83]. 
Coleman  and  Andrews  [15]  used  unsupervised  clustering  on  the  multi- 
dimensional feature  space  to  segment  an  Image.  An  alternative  to  the 
traditional  pixel  classification  approach,  which  makes  use  of  the 
spatial  coherence  of  uniformly  textured  regions,  has  been  considered  by 
Davis  and  Mitchie  [19].  In  the  area  of  general  scene  analysis,  Nevatia 
[61]  has  developed  a  technique  for  isolating  edges  of  non-textured 
objects  from  a  textured  background.  More  recently  research  has  been 
directed  towards  texture  segmentation  in  the  framework  of  estimation 
theory.  Chen  and  Pavlidis  [12,13]  have  combined  statistical  parameter 
estimation  with  a  hierarchical  segmentation  scheme.  Hansen  and  Elliott 
[31]  have  proposed  a  method  which  uses  Markov  field  models  for 
segmentation  of  noisy  two-intensity-level  Images.  This  technique  can  be 
easily  extended  to  texture  segmentation. 
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The  method  proposed  in  this  work  for  segmenting  arbitrary  views  of 
multi -surface  textured  scenes  differs  in  may  aspects  from  available 
techniques  for  segmenting  frontal  views  of  textured  surfaces.  Complex- 
ities arising  from  perspective  transformation,  inherently  associated 
with  an  image,  are  incorporated  into  the  segmentation  operator. 
Boundaries  in  an  image  are  attributed  to  physical  edges  delimiting 
surfaces  of  different  orientation  or  different  reflectance. 

The  method  is  aimed  at  a  class  of  textures  which  are  statistically 
homogeneous.  In  terms  of  the  proposed  texture  taxonomy,  categories  1-3 
are  considered.  No  a  priori  knowledge  of  the  number  and  the  character- 
istics of  textured  surfaces  appearing  in  a  scene  is  required.  It  is 
assumed  that  the  scene  is  lighted  by  a  single  distant  light  source  and 
that  the  effects  of  shadows  and  specular  components  are  insignificant. 
A  set  of  constraints,  emerging  from  the  underlying  imaging  model  and 
properties  of  textures  being  considered,  is  initially  used  to  identify 
the  uniform  regions.  A  uniform  region  is  a  region  in  an  image  that 
originates  from  a  planar  surface,  containing  statistically  homogeneous 
texture.  The  method  used  to  identify  the  uniform  regions  is  described 
in  section  5.1.  Upon  identifying  the  uniform  regions,  estimation  of  the 
boundary,  located  in  the  intermediate  zone  between  two  uniform  regions, 
is  reduced  to  the  problem  of  determining  the  boundary  between  two 
regions  of  known  properties.  Two  methods  of  performing  this  task  are 
considered  in  sections  5.2  and  5.3  respectively. 


•1 


77 

5.1    Isolation  of  the  Uniform  Regions 

Due  to  the  loss  of  homogeneity  Intrinsic  to  the  frontal  view  of  a 
texture  pattern,  one  of  the  major  difficulties  encountered  In  this 
research,  is  the  identification  of  the  regions  in  an  image  that 
correspond  to  a  planar  surface  in  the  3-D  scene.  A  change  in  the 
surface  orientation  or  a  change  in  reflectance  properties  may  result 
into  a  change  in  the  characteristic  angle.  On  the  other  hand, 
discontinuities  in  a  distance  from  the  viewer  or  a  change  in  surface 
reflectance  may  result  into  abrupt  changes  in  the  characteristic 
periodicity  measured  along  consecutive  characteristic  lines.  Finally,  a 
change  in  the  surface  orientation  may  cause  an  abrupt  change  in  the 
gradient  of  the  characteristic  periodicity  (measured  along  consecutive 
characteristic  lines). 

The  Isolation  of  uniform  regions  is  a  two-step  procedure  developed 
in  the  framework  of  a  merge-and-split  algorithm.  The  criterion  for 
region  merging  is  no  change  in  the  characteristic  angle  while  the 
criterion  for  region  splitting  is  the  existence  of  discontinuities  in 
the  characteristic  periodicity. 

The  overall  strategy  for  identifying  uniform  regions  is  described 
in  the  diagram  shown  in  Figure  20.  The  image  is  divided  into  K  non- 
overlapping  windows  and  the  characteristic  angle  in  each  one  of  them  is 
determined.  Any  of  the  possible  eight  neighboring  windows  W,-  is  merged 
with  if  n^-6<nj<jj^.+6  where  6  is  a  small  positive  number; 
are  the  characteristic  angles.  Gradual  merging  of  windows  results  in 
regions  that  may  have  originated  from  a  single  surface  in  a  scene. 

Windows  that  do  not  satisfy  the  merging  criterion  separate  two  uniform 
regions  and  form  a  region  of  ambiguity  in  which  the  boundary  is  located. 
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Image  I 


Division  of  I  into  adjacent  non-over- 
lapping windows  W^.  (m  x  m) 


Detenni nation  of  the  characteristic 
angle  Q.  for  W^.i=l  ,2,...  ,K 

Wi-»S- 


Merging  of  windows  W.  and  W.  based 
on  the  proximity  criterion  '^f  n^=n. 

I"*^Rl      *  *  •  •  »^|^  '  *  * '  ^ 


Determination  of  the  characteristic 
periodicity  ^or  each  R.  and  region 
splitting  if  appropriate 

R...r(])  .  r(2) 


Figure  20.    Determination  of  the  uniform  regions. 
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It  is  clear  that  the  reliable  identification  of  the  characteristic 
angles  is  of  the  foremost  importance  for  identifying  the  uniform 
regions.  In  practice,  as  we  have  already  pointed  out,  determination  of 
the  characteristic  angle  based  on  the  Fourier  power  spectrum  is  very 
difficult  due  to  texture  directionality  effects.  The  distinction 
between  the  peaks  due  to  the  directionality  of  texture  and  the  peak 
associated  with  the  characteristic  angle  is  not  always  straightforward, 
especially  in  view  of  the  effects  of  the  window  size  on  the  measured 
spectrum.  Our  experience  shows  that  the  procedure  discussed  in  section 
4.1.1.  is  not  necessary  for  obtaining  uniform  regions;  instead,  it  suf- 
fices to  utilize  the  pattern  vector  PY.=((P^^^<t)^^^^),(P^^^  , 
■t-,-^^^)....,  (P^'^^'l)^^'^^))  where  ^.^^^  ,  j=l,...,k,  are  the  angles 
corresponding  to  all  significant  peaks  in  the  angular  distribution  of 
the  Fourier  power  spectrum  for  window  ((t)^ ^''*^^>(ti^. ^"^^ )  and  merge  two 
windows  and  Wj  if  4.^. ^"'^=(t>j^"'^+6,  m=l,2,...,k  (in  our  implementation 
6=0).  This  approach  is  highly  reliable  and  does  not  introduce  addi- 
tional computational  burden  and  memory  requirements.  Figure  21(b)  shows 
the  identified  uniform  regions  for  the  image  in  Figure  21(a). 

Upon  determining  uniform  regions  based  on  the  characteristic  angle, 
characteristic  periodicities  are  evaluated  in  predetermined  intervals  in 
each  uniform  region  using  the  method  described  in  section  4.2.  In  this 
way  image  segmentation  is  embedded  into  the  determination  of  surface 
orientation.  Determination  of  the  boundary  location  between  two  regions 
having  the  same  characteristic  angle  is  straightforward.  The  boundary 
lies  at  the  point  where  characteristic  periodicity  or  its  gradient 
abruptly  changes.  Moreover,  the  boundary  is  actually  a  segment  of  the 
characteristic  line  at  that  point  [8].     On  the  other  hand,   if  two 
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(b) 


gure  21 —Continued. 
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adjacent  regions  do  not  share  the  same  characteristic  angle  the  boundary 
orientation  is  not  known  beforehand  and  has  to  be  determined  utilizing 
one  of  the  methods  described  in  sections  5.2  and  5.3.  Naturally,  the 
boundary  is  a  straight  line  since  it  originates  from  the  intersection  of 
two  planes. 

It  is  apparent  that  the  successful  completion  of  the  segmentation 
task  relies  on  the  choice  of  the  window  size  used  for  determining 
uniform  regions.  This  work  resulted  in  the  development  of  an  algorithm 
for  the  automatic  selection  of  the  window  size  to  be  used  for  initial 
labeling.  The  window  size  selection  is  performed  under  the  assumption 
that  all  textures  in  the  image  at  hand  contain  texels  of  comparable 
sizes;  i.e.  the  image  does  not  contain  a  combination  of  micro  and  macro 
textures. 

The  window  size  selection  is  performed  by  determining  Fourier  power 

spectra  for  a  sequence  of  eccentric  windows  of  sizes  (2^*a)x(2^*a) 

i=0,l,...    where  a  is  the  starting  window  size  (in  this  implementation 

a=4  pixels).      The  pattern  vector    ^r^=('l>^^^^(t)^^^^...<t)^.^'^h  .  where 

j=1.2...k    represent  the  angles  corresponding  to  all  significant 

peaks  in  the  angular  distribution  of  the  Fourier  power  spectrum  for 

and     <t>^.^^'*'^^><t'^.^^^  ,    is    determined    for   each   of   the   windows.  A 

particular  window  W_  Is  chosen  for  initial  labeling  if    it  =Tr    ,  while 

m  ^  m  m+1 

^m^'^m-l*  measurements  are  performed  in  an  arbitrary  image  region 

and  if  the  pattern  vector    ttj^^  is  different  from  upon  reaching 

window  sizeV4[N  x  N)  where  N  x  N  is  the  image  size,  similar  measurements 

are  attempted  in  another  region.    Examples  of  the  visual  changes  in  the 

Fourier  power  spectrum  for  the  image  shown  in  Figure  22(a)  are  shown  in 

Figure  22(b).  Window  of  64x64  pixels  is  chosen  for  initial  labeling  in 
this  case. 
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16x16 

Figure  22.    Determination  of  the  initial  window,  (a)  Image  sample, 
(b)  Power  spectrum. 
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64x64 

Figure  22-Continued. 
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128x128 
(b) 


Figure  22— Continued. 
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5,2    Determination  of  the  Boundary  Location 
by  Least  Squares  Curve  Fitting 

Determination  of  the  uniform  regins  results  in  connected  regions 
that  have  originated  from  planar  surfaces  in  a  scene  and  the  inter- 
mediate regions  exhibiting  different  properties.  In  this  section  we 
present  a  method  of  determining  the  boundary  location  by  utilizing  least 
squares  curve  fitting  to  a  predetermined  number  of  estimated  boundary 
points.  The  underlying  idea  in  determining  individual  boundary  points 
is  best  understood  using  the  example  shown  in  Figure  23.  Having 
identified  the  uniform  regions  R^-  and  Rj,  their  characteristic  angles 
.  and  the  intermediate  region  Ry,  a  subregion  of  R^-  is 
characterized  by  its  6-matrix 

^|J.a^=f9lv^       C.v=0,l....,k-1  ; 

subregion        is  chosen  as  shown  in  Figure  23  and  has  size  QxR,  Q«R  and 
typically  Q=l.     The  element    gj^^    is  the  normalized  frequency  with 
which  pairs  of  pixels  whose  distance  is  given  by  vector    v={d  cosn^, 
d  sinn^)  ,  having  intensities    5  and  v   appear;  k  is  the  number  of  gray 
levels.    The  associated  inertia  measure 

has  minima  for 


=1  2 
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where  is  the  characteristic  periodicity.     Since    n.*^.  G"^|^  ^ 

does  not  have  minima  for  d=Ti^  ,  The  task  of  the  boundary  estimator  is 
to  minimize 


where        „   and      „    are  elements  of  „  _  respec- 

ts,v       ^c.v  h^,"j 

tively,  g"p  „  is  an  element  of  G?  „  of  the  intermediate  region  of 
dimensions  QxM  and    aj(C,v)    is  the  subregion  of  S^^  which  belongs  to 

(aJ(?,v)+aJ(5,v)=M)  .      Upon   estimating     aj{c,v)  for  ?,v=0,l,...,k-l 
using  equation  (5.1)  and  calculating  its  average  value 


the  boundary  point  is  estimated  to  be  at  location  (x,y)  where 


x=H  -X^COSS^j 

0  i 


y=VQ-x^sinn^ 


and  Hq.Vq  are  as  indicated  by  Figure  23.  Upon  initially  estimating  the 
location  of  an  individual  boundary  point  the  procedure  is  repeated  for 
the  same  Intermediate  region  extended  into  subregion  by  a  known 
amount  so  that  the  dimensions  of  are  Qx(M+a^)  .  The  minimiza- 
tion procedure  with  the  new  values  of  the  G-matrix  for  the  intermediate 
region  results  into  a  new  value  for  x  .  By  performing  this  procedure 
iteratlvely  one  can  determine  the  boundary  location  with  a  specified 
accuracy  e.    The  iteration  stops  when 
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-  a]*  -  a^l  <   t  ,  (5.2) 


If  the  accuracy  criterion  is  not  satisfied  after  a  particular  number  of 
extensions,  Sy  is  considered  atypical  for  the  texture  pattern  and  a  new 
subregion  is  chosen  for  the  same  procedure.  In  this  way,  we  obtain 
individual  boundary  points  which  are  used  for  the  least  squares  boundary 
fitting  whereby  the  perpendicular  distance  from  the  points  to  the 
boundary  is  minimized.  Naturally,  boundary  is  assumed  to  be  a  line 
segment  since  it  originates  from  the  intersection  of  two  planar 
surfaces. 

The  boundary  is  fitted  to  the  determined  boundary  points  using  the 
least  squares  approximation.  The  G-matrix  has  been  implemented  as  an  5 
X  5  array.  The  value  of  e  in  equation  (5.2)  has  been  chosen  e=.l  max 
Cg^       .    Figures  24-26  show  the  results  obtained  by  using  this  method. 

The  cube  shown  in  Figure  24  has  been  sampled  as  a  220  x  220  data 
array.  The  initial  window  size  used  to  identify  uniform  regions  was  32 
X  32.  Isolation  of  the  uniform  regions  has  been  finalized  based  on  the 
characteristic  angle.  The  boundaries  obtained  are  very  satisfactory; 
the  slight  discrepancies  are  attributed  to  the  relatively  blurred  image 
and  the  resulting  difficulty  in  separating  the  object  from  the 
background. 

The  picture  in  Figure  25  has  been  digitized  as  a  200  x  200  array. 
Since  the  surface  planes  are  parallel,  the  segmentation  is  based  on  the 
abrupt  changes  in  the  characteristic  periodicity.  Irregularities  in  the 
brick  pattern  and  the  presence  of  a  tree  are  thought  to  have  caused  the 
minor  translation  of  the  boundaries. 
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Figure  24.    An  example  of  three  textured  surfaces  of  different 
orientation. 


91 


Figure  25.    An  example  of  four  parallel  textured  surfaces. 
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The  scene  shown  in  Figure  26  is  relatively  more  complex.  Identi- 
fication of  the  uniform  regions  involved  region  merging  based  on  the 
characteristic  angle  and  region  splitting  based  on  characteristic 
periodicity.  The  object's  surface  is  assumed  locally  planar  in  this 
case. 

5.3    Linear  Recursive  Filter 

The  linear  recursive  filter  (LRF),  otherwise  known  as  the  Kalman 
filter,  has  as  its  developmental  roots  the  concept  of  least  squares 
estimation  and  the  ideas  of  Wiener  and  Kolmagorov  [1,23,41,42]. 
Essentially,  the  current  state  estimate  is  a  function  of  previous 
estimate  and  the  new  measurement  data  sample.  The  recursive  nature  of 
the  filter  and  the  fact  that  it  requires  storage  of  only  the  last 
estimate  makes  it  very  appealing  for  computer  implementation. 

The  LRF  equations  are  derived  by  assuming  a  model  of  the  system 
wherein  the  state  vector     evolves  according  to  the  equation 

h'\-l  h-l"^  (5.3) 

where  is  an  N  x  N  state  transition  matrix  and  Wj^  is  the  process 

noise  such  that 


E{w^}=0  .  E{w^w{}=Q^6 


kl 


where  E  denotes  the  expectation  operator  and  is  the  Kronecker's 
delta.      It  is  assumed  that  M  simultaneous  measurements  are  made  at 


instance  k  such  that 
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>  0  . 


The  filter  computations  are  performed  In  two  stages, 


Propagation  stage 


Pk^-^^Vl^k-l^^^Vl^Vl  (5.6) 
Update  stage 

x,^(+)=x,^(-)+K^[z^-H^Xj^(-)]  (5.8) 

''k(*^'^k('^"'^k"k''k("^  •  (5.9) 
The  variables  appearing  In  equations  (5.5)-(5.9)  denote 

X|^(-)        =      state   vector   estimate   at     c=C|^     given  measurements 
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P,(-) 


h 

R,. 


state  vector  estimate  at  ^^^i^.j  given  measurements 
ZpZ2,...,Z|^_^,(nxl); 

covariance  matrix  of  the  estimation  error  in 
Xj^(-),(nxn); 

covariance     matrix     of     the     estimation     error  in 
x,j_l(+),{nxn); 
measurement  vector  for  c=?|^,(mxl); 
system  transition  matrix,  (nxn); 
measurement  matrix,  (mxn); 
process  noise  covariance  matrix  at  c=C|(_i; 
measurement  noise  covariance  matrix  at  (;=C|^,(mxm); 
Kalman  Gain  Matrix  at  c=?|j,(nxm). 


Upon  properly  initializing  the  equations  and  propagating  the  states 
using  equations  (5.5)  and  (5.6)  the  update  operations  consists  of 
computing  the  gain  matrix  equation  (5,7)  and  using  it  to  obtain 
estimates  of  the  states  (equation  (5.8))  as  well  as  the  error  covariance 
matrix  equation  (5.9). 

In  this  work  we  have  utilized  a  system  and  measurement  model  which 
allow  estimation  of  the  boundaries  which  can  be  adequately  approximated 
by  a  curve  of  the  form 

y'agX^+a^x+aQ  . 


The  objective  is  then  to  estimate    a2'*l»^0  utilizing  a  set  of 

measurements   performed   in  the  intermediate  zone  between  two  uniform 

regions  R^.  and  R.  .  By  virtue  of  the  linearity  of  the  intermediate 
zone  the  measurement  for  subregions  y=const.  is 
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^  P^5+(p-5)Pj 
P 


(5.10) 


where  p^-  and  pj  are  the  properties  of  regions  R^-  and  Rj  respectively 
and  6  and  p  are  as  indicated  in  Figure  27.  The  quantity  z  can  be  easily 
expressed  in  terms  of  aQ,ai,a2  as 


(P^-PJ  2 

— : — — (a  2b  ^+a  jb+a  Q )  +p  ^ 


(5.11) 


where  b  is  as  shown  in  Figure  27. 

Diagonal  elements  of  the  G-matrices  of  regions  R^  and  Rj  are 
utilized  as  properties  p^-  and  pj.  Experimentation  with  specially 
designed  images  which  allow  comparison  between  the  obtained  measurements 
and  the  theoretically  expected  values  revealed  that  z  is  contaminated  by 
noise  n  which  can  adequately  be  modeled  as  the  sum  of  a  bias  and  a  zero- 
mean  white  random  process,  i.e. 


ri=B+v 


where  the  bias  B  is  a  random  constant  and  v  is  white  noise.  Accord- 
ingly, the  measurement  process  is  modeled  as 


z=z+B+v 


(5.12) 


By  defining  z=z-Pj  and  by  combining  equations  (5.11)  and  (5.12) 
one  obtains 


z=[-! — I  b     —  I  b   —  I  1] 

p  P  p 


+  V  . 


(5.13) 


Figure  27.    Measurement  scheme. 


98 


In  the  actual  implementation  we  have  utilized  three  simultaneous 
measurements     along     the    strips      y=b,y=h/3+b  and  y=(2/3)h+b,(Xy4  h, 
(Xx«  p,Oi b<;  h/3    (Figure  27).    Therefore,  the  measurement  equation  takes 
the  form 


where 


(5.14) 


(5.15) 


and 


v^=Cv^.V2.V3] 


Pi"Pi  2 


p 

Pi"Pj  2 
-I — i  b2 

p 


P 

Pi'Pj 

p 

Pi-Pj 


Pi-Pj 

p 

Pi-Pj 

p 

Pi-Pj 


(5.16) 


(5.17) 


bj=b,  b2=h/3+b  and  b2={2/3)h+b  and  the  state  vector 


(5.18 


The  implementation  of  LRF  requires  the  specifications  of  the 
process  noise  covariance  matrix  (Q),  measurement  noise  covariance  matrix 
(R),  covariance  matrix  of  the  estimation  error  (P^)  and  state  vector 

A 

estimate    (x  )  .    Proper  choice  of  values  for  the  elements  of  Q  and  R  is 
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crucial.  In  general,  it  is  necessary  to  choose  "conservative"  but  no 
too  large  values  for  these  quantities.  Large  elements  of  R  in  effect 
weight  the  measurements  very  little  and  coupled  with  the  bad  estimates 
for  result  into  unsatisfactory  image  segmentation.     On  the  other 

hand,  the  large  values  for  Q  cause  no  change  in  the  elements  of  the 
CO variance  matrix  of  the  estimation  error.  Choices  of  Q=0  or  R=0  are 
usually  Inappropriate  and  can  cause  serious  numerical  difficulties.  The 
correct  choice  of  Pq  is  not  as  important  since  its  effects  are  being 
washed  out.  In  our  experience,  the  initial  values  for  the  state 
variable  x^  have  little  effect  on  the  final  result  if  Q  and  R  are  well 
chosen.  The  particular  values  for  elements  of  Q,R,Pq  and  x^  ,  for  the 
purpose  of  this  work,  were  determined  empirically  from  preliminary 
observation  data.  An  example  of  the  obtained  result  is  shown  in  Figure 
28. 

The  LRF  is  ideally  suited  for  applications  where  real-time  boundary 
estimation  is  required  and  little  memory  is  available.  Moreover,  it 
provides  an  insight  into  the  uncertainty  of  the  obtained  boundary 
location  through  the  estimation  error  covariance  matrix.  However,  it 
calls  for  the  knowledge  of  the  statistical  characteristics  of  noise  and 
hence  requires,  on  the  part  of  the  user,  deep  understanding  of  the 
properties  of  the  class  of  images  under  consideration. 


Figure  28.    An  example  of  a  segmented  image  obtained  by  using 
Linear  Recursive  Filter. 


CHAPTER  6 

SURFACE  ORIENTATION  FROM  ROTATIONAL  STEREO 


In  this  chapter  we  propose  a  method  which  utilizes  rotational 
stereo  together  with  textural  information  as  the  means  of  estimating 
surface  orientation  more  accurately.  The  two  views  of  a  scene 
constitute  rotational  stereo  pair  when  the  camera  is  rotated  between  two 
exposures.  The  method  is  highly  advantageous  for  the  class  of  images 
considered  since  it  makes  use  of  two  images  without  requiring  the 
establishment  of  pair-wise  correspondence  between  individual  points  in 
the  two  images.  In  comparison  to  the  case  where  a  monocular  view  of 
textured  surfaces  is  available  (chapter  3)  the  method  yields 
significantly  more  accurate  results. 

The  general  principles  of  using  two  images  of  a  scene  are 
summarized  in  section  6.1.  Section  6.2  concentrates  on  the  development 
of  the  computational  procedure  for  the  proposed  rotational  stereo. 
Implementation  issues  and  results  are  discussed  in  section  6.3. 

6.1    Computational  Theory  of  Stereopsis 

The  fact  that  a  human  deduces  Information  on  the  relative  depth  of 
visible  surfaces  from  the  slight  disparities  in  the  images  received  by 
the  left  and  right  eyes  is  well  established.  Consequently,  a  number  of 
researchers  [4,51,52,54,59,60,85]  have  advocated  that  stereopsis  is  a 
viable  approach  to  computer  vision.     In  particular,  stereo  images  have 

found  application  in  robot  systems  [39,60]. 
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Given  two  images  Ij  and  I2  of  a  scene  (Figure  29)  the  theory  or 
stereopsis  involves  identification  of  points  Pj^  in  li  and  ?2  ^2  ^^i^h 
are  the  projections  of  a  point  P  in  a  3-D  scene.  The  coordinates  of  P 
are  then  determined  based  on  the  coordinates  of  Pj^  and  P2*  geometrical 
considerations  arising  from  underlying  perspective  projection  and  the 
camera  parameters.  Despite  the  fact  that  the  procedure  involves  very 
simple  equations  the  application  of  stereopsis  in  computer  vision  has 
been  limited  primarily  due  to  the  fact  that  it  requires  the  establish- 
ment of  correspondence  between  points  in  two  images  which  is  recognized 
to  be  inherently  difficult. 

An  attempt  to  solve  the  correspondence  problem  can  be  made  under 
the  assumptions  that  (1)  a  given  point  on  a  physical  surface  has  a 
unique  position  in  space  at  any  one  instance  of  time,  and  (2)  matter  is 
cohesive,  i.e.  the  surfaces  of  the  objects  are  generally  smooth  and 
discontinuities  in  distance  due  to  roughness  and  cracks  are  negligible 
relative  to  the  distance  from  the  viewer.  Two  approaches,  the  area 
correlation  and  the  interest  operator  based  stereo,  have  been  proposed 
as  a  means  of  solving  the  correspondence  problem.  The  methods  that 
utilize  area  correlation  operate  on  a  succession  of  windows,  i.e.  image 
subregions,  and  search  for  the  window  in  one  Image,  e.g.  Ij,  that  is 
optimally  correlated  to  a  window  in  The  Interest  operators,  on  the 
other  hand,  correlate  selected  points  In  two  images,  e.g.  zero-crossings 
[53].  Utilization  of  the  points  of  Interest  is  computationally  more 
efficient  since  the  actual  matching  is  reduced  to  fewer  points. 

Neither  of  the  two  approaches  is  convenient  for  the  problem  at 
hand.  Area  correlation  is  prohibitively  expensive  since  its  implemen- 
tation requires  N^m^  elementary  operations  for  the  m  x  m  window  and  N  x 
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Figure  29.    Stereo  imaging. 
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N  Image.  The  proper  selection  of  the  points  of  interest  in  texture  is  a 
very  difficult  and  computationally  expensive  procedure  as  well. 
Instead,  we  propose  a  different  type  of  stereo  Images  and  the  method 
that  does  not  require  matching  between  the  individual  points. 


For  simplicity  the  idea  of  rotational  stereo  is  explained  at  first 
for  a  scene  consisting  of  a  planar  surface  containing  an  ideal  texture 


categories  of  texture  is  then  considered. 

Given    two    Images,      1^:  aj^x+bj^y+Cj^z+dj=0      and  agX+bgy* 
C2Z+d2=0  ,  of  a  textured  surface,    S:  Ax+By+Cz+D=0  ,  which  are  rotated 
relative  to  each  other  (Figure  30)  and  their  local  coordinate  systems 
x^y^Zj  and  the  characteristic  vector        ,  i.e.  the  unit  vector 

lying  along  the  characteristic  line   L„  is 


(the  choice  of  the  direction  along  the  characteristic  line  is  Immaterial 
for  our  purposes).  Similarly,  the  characteristic  vector  ^2  ^"  X2 
y2Z2,  coordinate  system  is  given  by 


6.2    Rotational  Stereo 


pattern. 


The    generalization    to    multi-surface    scenes    and  other 


1 


(6.1) 


(6.2) 


The  transformation  matrix  relating  coordinates  in  Xj^yj^z^  to  the 
coordinates  in  X2y2Z2 
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Figure  30.    Rotational  stereo  pair  of  images. 
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T  = 


Pi  gi  si 

P2  92  S2 
P3  93  ^3 


(6.3) 


where  p^.g^.s^;  P2,g2.S2;  P3»93»S3  aif*©  the  direction  cosines  of 
^I'^r^l  relative  to  axes    X2,y2»Z2  •         using  equation  (6.3) 

equation  (6.2)  can  be  written  in  the   ^i^i^i   coordinate  system  as 

^2  =  (Pini2+P2n2)T'i+(gim2+g2n2)ji+(Sjm2+S2n2)itj  .  (6.4) 


Since  the  characteristic  lines  and  L^^  belong  to  the  two  families 
of  lines  lying  in  S,  the  vector  product 


^1^^2  '  ^itni(Sim2+S2n2)]  +  jjC-m^ (Sjm2+S2n2)] 
+  ICjCmj(gjm2+g2"2^  "  nj(pjm2+P2n2)] 


(6.5) 


determines  the  surface  normal.  From  equation  (6.5)  the  surface 
parameters  are 


A  =  nj(Sj^m2+S2n2) 
B  =  -m^ (Sjm2+S2n2) 


C  =  mj(gjm2+g2"2^'"l^Pl"'2*P2"2^ 


(6.6) 
(6.7) 
(6.8) 


or,  in  terms  of  surface  tilt  and  slant 


T  =  tan"^(-'"l/nj)±Tr/2 


(6.9) 


=  tan'-^((Sjm2+S2n2)(nj^+m^^j'^2) 


(6.10) 
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The  notion  of  the  characteristic  angle  can  be  extended,  as  we  have 
already  pointed  out,  to  categories  of  statistically  homogeneous  natural 
textures  as  well  as  to  some  types  of  irregular  textures.  Moreover,  the 
procedure  is  useful  for  the  multi -surface  multi -textured  scenes. 
However,  identification  of  the  uniform  regions,  in  this  case,  must 
precede  determination  of  the  surface  orientation  since  it  is  necessary 
to  globally  match  the  textured  surfaces.  Identification  of  the  uniform 
regions  has  been  implemented  in  the  framework  of  the  merge-and-split 
algorithm  described  in  detail  in  chapter  5. 

6,3    Implementation  and  Results 

Implementation  of  the  proposed  method  assumes  precise  knowledge  of 
the  rotational  parameters  (equation  (6,3)),  Since  the  camera  control 
equipment  was  hand-made  and  no  special  control  system,  such  as  one 
manufactured  by  Cohu  Inc.  [30],  was  available,  for  the  purpose  of 
performing  accurate  measurements  the  camera  was  rotated  only  about  the 
xj  or  yi  coordinate  axes.  In  order  to  ensure  that  ti^x^^*^  three 
images  of  a  scene,  1^,  I2  (rotated  about  Xi)and  I2  (rotated  abot  yj), 
were  obtained,  but  only  two  of  the  images,  which  satisfy  the  stated 
conditions,  were  used  in  the  analysis. 

In  terms  of  the  transformation  matrix  T,  rotation  about  the  x^- 
axis,  for  an  angle  6  (Figure  31),  is  described  by 

10  0 
T  =  [0    cos  6  sin  8]  . 
0  -sin  B  cos  6 

The  characteristic  vector    ^,  is 
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Figure  31.    Rotation  about  the  x,-axis. 
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The  characteristic  vector         equation  (6,4)  is 

^2  "  "^^l"*"  3n2jj+{sine)n2i<j 
and  the  vector  product  x 

^1^^2  ^  +(sin6)nj^n2Tj+(-sinS)n2mj^jj+  (iHj^ng  cos  Q-n^m^^)^^  . 

Hence, 


while 


A  =  +"112  ^  (6.11) 
B  =  -n2ni^  sin  B  (6.12) 
C  =  nij^ng  cos  6-nj^m2  (6.13) 


T  =  tan"^(-"'l/n^)±Tr/2  (6.14) 
a  =  tan'^(sin  6  n2 (nj^+m^^ .  (6.15) 


In  the  case  of  rotation  about  the  y^-axis  for  an  angle  a  (Figure 
32)  the  transformation  matrix  takes  form 


cos  a  0  -sin  a 

T  =  [  0      1      0    ]  (6.16) 
sin  a  0    cos  a 


and  the  characteristic  vector 


no 


Ill 


^2  =  (cos  a       itj^+n^ Jj^+(-s1n  a  m^)^^  . 
Equation  (6.5)  then  becomes 

^j^x^2""i"'3("Sln  a)f j^-Hnj^m^  s1n  a  jj^+dn^n^  -  cos  a  nj^m^)!^^ 

and 

A  =  -n^m^  s1n  a 

B  =  ^1^2  " 
C  =  m^n^  -  cos  an^m 

OP 


(6.20) 
(6.21) 

Implementation  of  the  rotational  stereo  requires  determination  of 
the  characteristic  angles    0^,^52(^3)  utilization  of  the  equations 

(6.11)-(6.13)  ({6.17)-(6.19))  or  alternatively  (6.14)  (6.15)  ((6.20) 
(6.21)).  The  characteristic  angles  are  determined  in  the  manner 
described  in  chapter  4. 

Figures  33(a)  and  33(b)  show  an  example  of  the  rotational  stereo 
pair  of  images.  Image  33(b)  was  obtained  by  rotating  the  camera  about 
xj-axis  for  angle  .2618  rad.  Two  of  the  three  visible  sides  of  the  cube 
are  considered  in  our  analysis.  The  top  side  is  not  sufficiently 
visible  for  our  purposes.    The  characteristic  angles  obtained  for  the 


(6.17) 
(6.18) 
(6.19) 


T  = 


tan"^(-"'l/n^)±Tr/2 
tan"^(s1n  a  mAn.^Ha^^f^^  . 
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Figure  33.    Rotational  stereo  pair  of  images. 


(b) 

Figure  33— Continued. 
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left   side   of   the  cube   (the  grid   pattern)   are     nj^'^'^l     ""^^  ^'"^ 

fi2L=2.02    rad  for  Figures  33(a)  and  33(b)  respectively.    For  the  right 

side  of  the  cube  (star  pattern)  the  characteristic  angles  are  nj^|^=1.32 

rad  and   ^21^=1. 10    rad.    Direct  application  of  equations  (6,11)  -  (6.13) 

yields  surface  parameters    A|_=.224,  B|^=.054,  C|_=.225    for  the  left  side 

of  the  cube  and    A|^=.222,  B(^=-.057,  C|^=-.225    for  the  right  side  of  the 

cube.         The     angle     between     the     two     sides     or     the     cube  is 

(|)=cos '  (—-=-.  -T=-)       =1.60      rad    where      !J,=A,  t",+B,  j.+C,  ic.  and 

^    \K\     iK\  1    L  1    L  1    L  1 

N2=Aj^ij+B|^3j+C|^kj^  .    Naturally,  the  actual  value  is    41=1.57    rad.  The 

accuracy  of  the  obtained  results  may  be  further  improved  by  enlarging 

the  set  of  angles  considered  in  determining  the  characteristic  angle 

(section  4.1.1). 


CHAPTER  7 
CONCLUSIONS 


The  aim  of  this  study  has  been  to  develop  a  computational  theory  of 
scene  description  from  textural  information  available  in  images.  The 
visible  surfaces  in  a  scene  are  represented  in  terms  of  planar  surfaces 
whose  orientations  are  described  by  their  normals.  Boundaries  arising 
from  changes  in  surface  orientation,  surface  reflectance,  and  abrupt 
changes  in  the  distance  from  the  viewer  are  identified.  No  a  priori 
knowledge  on  the  scene  is  assumed  except  that  the  frontal  views  of 
textured  surfaces  belong  to  ideal,  regular  or  globally  regular 
textures.  Perspective  projection  has  been  chosen  as  the  appropriate 
imaging  model. 
Surface  Orientation 

Under  the  assumptions  adopted  in  this  work  a  single  view  of  a  scene 

does  not  provide  sufficient  information  to  accurately  determine  surface 

orientation.     Instead,  we  have  proposed  a  two-step  procedure  whereby 

surface  tilt  is  determined  and  surface  slant  estimated  from  measurements 

in  an  image.    Two  views  of  a  scene,  which  constitute  a  rotational  stereo 

pair,   have  been  utilized  as  a  means  of  more  accurately  determining 

surface  orientation.      The  main  advantages  of  the  rotational  stereo 

approach  are  its  simplicity  and  its  higher  accuracy  in  comparison  to  the 

case  where  only  a  monocular  view  of  textured  surfaces  is  available. 

Moreover,  the  method  does  not  require  the  difficult  problem  of  matching 

between  the  individual  points  in  two  stereo  Images. 
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The  Implementation  of  the  proposed  method  was  attempted  both  in  the 
transform  (Fourier  domain)  and  spatial  domain.  Based  on  experimentation 
with  different  images  we  have  reached  the  following  conclusions: 

(1)  Fourier  domain  analysis  is  superior  to  the  spatial  domain 
analysis  for  globally  regular  textures  since  it  is  immune  to  the 
presence  of  moderate  amounts  of  noise  (in  the  form  of  texture 
irregularity,  blur,  shading,  etc.).  2-D  spectral  analysis  is 
particularly  well  suited  for  the  initial  labeling  since  it  does  not 
require  a  priori  knowledge  on  the  scene.  The  major  shortcoming  of  2-D 
spectral  analysis  is  its  computational  complexity. 

(2)  Interpretation  of  the  1-D  Fourier  spectrum  is  straightforward 
and  the  method  is  computationally  very  efficient.  However,  its 
utilization  calls  for  specially  designed  hardware  or  software  for  image 
rotation. 

(3)  Spatial  domain  analysis  is  advantageous  in  the  case  of 
aperiodic  textures  whose  elements  differ  in  shape  and  slightly  in  size 
since  it  allows  for  simultaneous  determination  of  the  characteristic 
angle  and  the  direction  of  texture  gradient. 

Segmentation 

A  set  of  constraints  arising  from  geometrical  considerations  and 
texture  categories  considered  (ideal,  regular,  and  globally  regular) 
have  been  utilized  to  identify  the  uniform  regions  in  an  image.  In  this 
way,  the  segmentation  problem  reduces  to  the  estimation  of  the  boundary 
location  in  the  intemediate  zone  between  two  uniform  regions.  Least 
squares  curve  fitting  and  a  linear  recursive  filter  have  been  employed 
to  accomplish  this  task. 
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The  measurement  scheme  (used  by  both  methods)  has  been  evaluated 
based  on  the  analysis  of  the  differences  between  the  expected  and 
obtained  measurements  on  a  set  of  images  specially  designed  for  this 
purpose.  The  evaluation  has  shown  that  the  linearity  assumed  in  the 
intermediate  zone,  together  with  the  choice  of  cooccurrence  matrices 
provides  good  measurements.  The  window  clipping  effects  which  result  in 
a  measurement  bias  (primarily  for  ideal  textures)  have  been  circumvented 
by  an  iterative  estimation  procedure  for  the  least  squares  method.  In 
the  case  of  the  Kalman  filter,  the  clipping  effects  have  been  overcome 
by  the  introduction  of  an  additional  state  which  represents  the 
measurement  bias. 

Both  segmentation  operators  are  very  effective  and  their 
applicability  goes  beyond  the  class  of  images  considered.  Least  squares 
curve  fitting  is  attractive  for  Its  simplicity  while  linear  recursive 
filters  are  of  particular  interest  for  applications  where  real-time 
processing  is  needed  and  limited  memory  is  available. 
Future  Work 

An  Immediate  extension  of  our  segmentation  operator  is  the 
utilization  of  the  optimal  linear  recursive  filter  for  the  piecewise 
boundary  estimation.  A  more  fundamental  extension  of  this  work  should 
be  sought  in  better  understanding  of  textures  in  general.  A  design  of 
the  structural  analyzer  would  be  of  great  importance  to  extending  this 
work  to  aperiodic  textures. 
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